The CHEOPS cluster

The CHEOPS cluster is a high-performance computing cluster with 9712 Intel Nehalem compute cores spread over 841 nodes, which we employ in a broad variety of number crunching projects.

CHEOPS resources

HPC Antrag to get a user account on CHEOPS
The CHEOPS primer / brief instructions

ALPS on CHEOPS

A system-wide ALPS build of version 2.1.1 is available at

/opt/rrzk/software/alps/2.1.1
Compiling ALPS on the CHEOPS cluster
- Download and unzip the ALPS sources, which creates a new ALPS source directory
  curl -O https://alps.comp-phys.org/static/software/releases/alps-2.1.1-r6176-src-with-boost.tar.gz
  tar -xzvf alps-2.1.1-r6176-src-with-boost.tar.gz
- Start an interactive job to host the compilation process, which will be quite taxing on the system resources and should better be kept off the log-in node (the following is a single line):
  salloc --cpus-per-task=4 --mem=4gb --time=02:00:00 --account=AG-Trebst srun -n 1 --pty bash
- Create and move to a build directory
  mkdir ~/alps-2.1.1-bld-with-boost
  cd ~/alps-2.1.1-bld-with-boost
- Load a couple of necessary modules via:
  
  module load intel/13.1 mkl/11.0 intelmpi/4.1.0 hdf5/1.8.11 szlib/2.1.0
- Call cmake to configure ALPS (the following is a single line):
  
  cmake28 -DBoost_ROOT_DIR:PATH=~/alps-2.1.1-r6176-src-with-boost/boost
  -DCMAKE_INSTALL_PREFIX=~/alps/2.1.1
  -DHDF5_LIBRARIES:FILEPATH=/opt/rrzk/lib/hdf5/1.8.11/lib/libhdf5.so
  -DHDF5_INCLUDE_DIR:PATH=/opt/rrzk/lib/hdf5/1.8.11/include
  -DSZIP_LIBRARIES:FILEPATH=/opt/rrzk/lib/szip/szip-2.1/lib/libsz.so
  -DSZIP_INCLUDE_DIR:PATH=/opt/rrzk/lib/szip/szip-2.1/include
  -DBLAS_LIBRARY=/opt/rrzk/compiler/intel/composer_xe_2013.5.192/mkl/lib/intel64/libmkl_rt.so
  -DLAPACK_LIBRARY=/opt/rrzk/compiler/intel/composer_xe_2013.5.192/mkl/lib/intel64/libmkl_rt.so
  -DFFTW_LIBRARIES:FILEPATH=/opt/rrzk/compiler/intel/composer_xe_2013.5.192/mkl/lib/intel64/libfftw3xc_intel.a
  -DFFTW_INCLUDE_DIR:PATH=/opt/rrzk/compiler/intel/composer_xe_2013.5.192/mkl/include/fftw
  -DALPS_BUILD_PYTHON=OFF
  ~/alps-2.1.1-r6176-src-with-boost/alps
- Compile the ALPS sources:
  
  make -j 4
  make test
  make install
- Congratulations! Your own ALPS build should now be available. The binaries are located in
  ~/alps/2.1.1/bin/

Job submission

An example submission script for an MPI job
#!/bin/bash -l
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=8
#SBATCH --ntasks=16
#SBATCH --mem=12gb
#SBATCH --time=01:00:00
#SBATCH --account=AG-Trebst

export WORKDIR=$HOME
export ALPSDIR=$HOME/alps/2.1.1
export OUTPUTDIR=$HOME/Jobs

cd $OUTPUTDIR
module load intel/13.1
module load mkl/11.0
module load intelmpi/4.1.0
module load hdf5/1.8.11
module load szlib/2.1.0

# output start time
echo start time
date
echo

# start MPI job
srun -n $SLURM_NTASKS $ALPSDIR/bin/spinmc --mpi $WORKDIR/scan.in.xml

# output finish time
echo finish time
date
echo
Submit it via
sbatch job.sh

Job management and queue status

We are collecting some short scripts to help manage your jobs and check the status of the queue: https://github.com/maxhgerlach/cheops-scripts

Where did the old computing website with legacy instructions go?

If you need to go back to one of our previous versions of this webpage go to
computing website January 2012 - October 2013

The JUWELS cluster @ FZ Jülich

The JUWELS cluster is a high-performance computing cluster with some 122,768 Intel Xeon Platinum 8168 compute cores spread over 2567 nodes, which is located at the Forschungszentrum Jülich. We are using this resource in many of our production runs.

JUWELS resources

User info to get started on JUWELS