Computational OPtimization Solutions cluster managed by MFCF University of Waterloo -- Waterloo, Ontario, Canada

Job Scheduler

The Moab suite by Adaptive Computing is a very sophisticated and powerful job scheduling system. It runs on the Linux head node but manages all the machines in the cluster.

It is possible to define very complicated rulesets to describe the rules by which the priority and scheduling of jobs are determined. In our case, though, we are using a very simple set-up in which all users have equal priority. Should the need arise in future, we can add rules to define different categories of users, or groups of users, and different conditions affecting relative priorities.

The Linux nodes run the Torque job scheduler, while the Windows nodes run a native Windows HPC Cluster scheduler. Moab understands both so you can submit jobs to either platform via Moab. If you submit jobs directly to Torque or Windows HPC, Moab can see them but does not manage them. If you run jobs outside of any job scheduler, merely launching them from the command line, Moab sees that the system is busy but has no control over it.

The simplest thing, then, is to learn to submit jobs to Moab on the Linux head node, linopt.math. For each job, you need to tell Moab how many CPUs (cores, not sockets) you need, how much wallclock time you expect the job to run in, and which operating system you want to use.

When enough machines are free to meet the job's requirements, Moab will reboot some machines to the specified OS if necessary, then run the job.

There's much to learn about this software system, more than is practical to document here. Refer also to
http://www.clusterresources.com/products/mwm/docs ,
especially
http://www.clusterresources.com/products/mwm/docs/4.6user-cmds.shtml .

Examples of job submission

1. Using Moab
   

   In this example, we run the command "sleep 300". This takes 300 seconds to complete, so we'll tell Moab the job needs 301 seconds of wall clock time to complete. We'll run it on Linux. The "msub" command submits a job to Moab. Moab prints the job identifier; in this example, moab.3570.

   linopt% echo "sleep 300" | msub -l walltime=301,os=linux
   moab.3570

   In the next example, we run the command "ping -n 100 localhost" as a trivial network test that does 100 pings, once per second, on a Windows node.

   linopt% echo "ping -n 100 localhost" | msub -l walltime=101,os=windows
   maob.3571

   The main point of these examples is to see how to tell Moab how much time a job needs and which OS you want to use.

2. Using Torque
   

   Torque runs only on the Linux nodes. You can submit a job directly to Torque using "qsub". Syntax is similar to "msub", but of course you do not specify an operating system. If there are no Linux nodes available, your job won't run. (If you used Moab instead of Torque, Moab would arrange for a Linux node to become available.)

   linopt% echo "sleep 300" | qsub -l walltime=301

3. Using Windows Powershell
   

   Powershell runs only on the Windows nodes. You can submit a job directly using "job submit" on the Windows head node. If there are no Linux nodes available, your job won't run. (If you used Moab instead of Powershell, Moab would arrange for a Windows node to become available.)

   winopt% job submit ping -n 100 localhost

4. Specifying processors and memory
   

   The foregoing examples merely ran a single job on a single CPU core. To run multiprocess jobs using multiple CPUs/cores, you need to tell Moab how many nodes (computers) you want, and how many processes per node you want. You can also specify how much memory you need. For example:

   msub -l nodes=8:ppn=4,pmem=1024mb,walltime=3600

   This job requires 8 nodes, each with 4 processors, with 1 GB of memory per process, and a runtime of an hour.

Examples of checking job queue status

This command shows the job identifiers, status, user, operating system, etc.:

linopt% mdiag -j

Examples of checking job status

Check the status of a job, using the job identifier that you got when starting the job, or from looking at mdiag -j:

linopt% checkjob moab.3570

Examples of checking machine status

To summarize the status of each of the nodes:

linopt% mdiag -n

This checks what's happening on node n003:

linopt% checknode n003

Switching to the other operating system

To reboot a node, say node n012, from Windows to Linux:

linopt% mnodectl -m os=linux n012

Use "os=windows" to boot from Linux to Windows.

Links to documentation

Moab:
http://www.clusterresources.com/products/mwm/docs
Windows HPC Cluster:
http://www.microsoft.com/hpc/en/us/default.aspx
Windows Powershell
http://www.microsoft.com/powershell
http://technet.microsoft.com/en-ca/library/ee221100.aspx

COPS Home | University of Waterloo.