![[UW logo]](http://www.uwaterloo.ca/UWlogo.gif)
Computational OPtimization Solutions cluster
University of Waterloo -- Waterloo, Ontario, Canada
![[MFCF logo]](http://www.math.uwaterloo.ca/mfcf/images/logo2.gif){kind=logo}
|
|
Computational OPtimization Solutions clusterUniversity of Waterloo -- Waterloo, Ontario, Canada |
|
Linopt.math and the Linux nodes are running the 64-bit Novell SuSE Linux Enterprise System version 11 operating system with Service Pack 1, SGI Foundation Software 2 Service Pack 1, and SGI ProPack 7 Service Pack 1.
Winopt.math and the Windows nodes are running Microsoft Windows HPC Cluster Server Edition 2008.
Skip past the Linux details to the Windows details if you like.
The SuSE Linux Enterprise System (SLES) includes all the basics, as well as such things as the GNU compilers, version 4.3.4. Release notes for SuSE are at /usr/share/doc/release-notes/. Use the uname(1) command to check the current kernel version.
A wide variety of additional SGI software products is installed. SGI Foundation includes tech support tools, InfiniBand enhancements, and system monitoring. SGI ProPack includes OS enhancements such as tuned math libraries and MPI libraries and tools.
The tuned math libraries come from a package called sgisolve. Experience on other SGI systems here suggests that significant performance gains can be achieved by using this library rather than generic math libraries. (See the News page for updates on this software, as it's not yet installed.)
See below for details on the SGI MPI suite.
Use the rpm -q -a (1M) command to get a complete list of both operating system and additional software products. The command rpm -ql packagename lists the files in package packagename. There's plenty of documentation at /usr/share/doc/packages/.
(Originally we had version 11.1.073 of the Intel Compiler Suite Professional Edition for Linux, which included everything except the MPI tools. We subsequently got upgraded to Intel Cluster Studio for Linux version 2011. If you compiled code using the earlier suite, you probably need to recompile.)
The Intel Cluster Studio for Linux, version 2011, is installed. This includes C, C++, Fortran 77, Fortran 90, debugger, Math Kernel Library (MKL), Threaded Building Blocks (TBB), Integrated Performance Primitives (IPP), and MPI parallel programming suite. Both IA-32 and Intel64 support are included. (The cluster's CPUs are Intel64 architecture.)
While the GNU C compiler (version 4.3.4) is available, the Intel suite will offer better performance because it takes advantage of the underlying hardware using Intel's knowledge of their own Xeon CPU. The MKL in particular is a library of well-tuned libraries for the popular numerical routines. You should see performance gains by using these rather than generic libraries.
To prepare your environment for the Intel development suite, you first need to determine whether your login shell is of the SH family (e.g. sh, bash) or the CSH family (e.g. csh, tcsh). You also need to choose an architecture to compile for, either "ia32" or "intel64". The native architecture of the cluster is intel64, but you can cross-compile for ia32 if you want.
Then run one of the following commands as appropriate for your shell and architecture (where ARCH is one of "ia32" or "intel64"):
Now your PATH will be set as needed. The C/C++ compiler command is icc. See man icc. The Fortran compiler command is ifort. See man ifort. The debugger command is idb. See man idb. (The GNU debugger is gdb.)
Intel Compiler documentation can be found under /opt/intel/composerxe-2011/Documentation/.
See below for information about the Intel MPI tools.
Third-party software is generally installed under /opt/.
Matlab 2012a is installed at /opt/matlab/R2012a/. Adjust your search rules to include /opt/matlab/R2012a/bin. As usual on research computers at UW, you must have paid for a personal Matlab licence in order for Matlab to work. You can use the parallel toolbox for parallel computation across the CPUs within one node. You can also use the Distributed Computing Server for parallel computation across nodes. (The previous version, 2010b, is installed at /opt/matlab/R2010b/ if you still need it.)
The R statistical software package, as well as the Python libraries SciPy and Numpy, plus dependencies such as Cython, gfortran, GSL, BLAS, LAPACK, ATLAS, and FFTW3 have been installed.
This is somewhat complicated because there are several different packages installed and you can use various compilers with them. In addition, for communication among nodes, you can choose between using the ordinary ethernet network or the higher-performance InfiniBand network. You'll need to include the appropriate libraries when compiling your code in order to choose the InfiniBand network. Refer to the MPI documentation for information about selecting InfiniBand instead of ethernet.
The default MPI suite (i.e. the one that comes with the system before buying any other third-party add-on versions) is the one that's part of SGI's Message Passing Toolkit (MPT). This based on the MPI 2.0 standard. The software is installed at /opt/sgi/mpt/mpt-2.01/ where, for example, there's a 'bin' directory containing the mpirun command. See man mpi for starters, as well as the release notes in the file /opt/sgi/mpt/mpt-2.01/doc/README.relnotes. A copy of the MPT User's Guide (in HTML and PDF format) is available at /usr/local/doc/SGI_MPT/. Some discussion of InfiniBand is included in that document.
The Intel Cluster Studio for Linux suite includes Intel's MPI, based on the MPI 2.1 standard. This is installed at /opt/intel/impi/4.0.1/ where there's a 'doc' directory with documentation including release notes, reference manual, and getting started guide. See the introductory information in the file /opt/intel/impi/4.0.1/README.txt as well.
To prepare your environment for using the Intel MPI tools, run one of the following commands according to your shell family. Unlike the compiler preparation commands, you don't specify the architecture as an argument of intel64 or ia32. You need to source a different file depending on which one you want.
The PBS job scheduler on the Linux systems includes MPI-related commands as well. See man pbs_mpirun for starters.
OpenMP is supported by the compilers. This uses built-in compiler directives, not a separate set of libraries like MPI. It lets you parallelize within a single machine, across the multiple CPUs/cores in that machine, using a much easier programming model than MPI. It does not allow you to parallelize across separate machines. See the man pages for the compilers. The most sophisticated use of the cluster in parallel would be to parallelize across machines using the InfiniBand network, and each job on each machine would in turn be an OpenMP job that parallelizes across the CPUs/cores in the machine.
The Intel development suite is installed on all the Windows nodes. Look in the Start menu, select Intel Development Suite, and you'll find several entries including on-line documentation.
Matlab 2010a is installed on all the Windows nodes.
Send feedback to:
COPS Home | University of Waterloo.
Last Updated 19 September 2012, by Robyn Landers