Myrinet Cluster HOW-TO

CTBP Myrinet cluster load for the last week
(The cluster is part of UCSD Rocks Grid.)

Cluster information

Available Myrinet enabled software

How to compile your application

Performance of the cluster

Cluster information

The CTBP Myrinet cluster (ctbp1m.ucsd.edu) consists of 32 Dell PowerEdge nodes each with two 2.8GHz Xeon processors and 2 GB RAM. The cluster uses Myrinet interconnects. This special networking provides low-latency and high throughput connection between nodes. To utilize the Myrinet network applications must be compiled with Myrinet-aware MPICH parallel library.

This cluster is designed for parallel jobs only, no single-processor jobs are allowed. If you have serial jobs please use the main CTBP cluster (ctbp1.ucsd.edu). Additionally, only Myrinet-enabled parallel applications can be used on the cluster to optimally utilize this resource. List of such applications installed on the cluster is below. If you have an application which is not on the list and would like to have an assistance in getting it compiled for the Myrinet cluster please send email to ctbp-help @ ctbp.ucsd.edu.

The Sun Grid Engine queuing system has been installed and configured on the cluster. All non-interactive jobs must be submitted through the SGE. See SGE How-To for more information.

Please note that all MPICH parallel jobs MUST be submitted using either /opt/mpich/myrinet/gnu/bin/mpirun or /opt/mpich/myrinet/intel/bin/mpirun. If you use any other version of mpirun your job will not use the Myrinet interconnects, it will use regular gigabit network for communication. Any such a job will be terminated.

Current SGE limits are:

Maximum job wall clock time: 24 hrs

Maximum number of processors per user: 16

Available Myrinet enabled software

How to compile your application

Use Intel 8.0 compilers (located in /opt/intel_fc_80 and /opt/intel_cc_80). Link it with Myrinet MPICH library in /opt/mpich/myrinet/intel. GM hardware level library is in /opt/gm/lib and include files in /opt/gm/include

You can also use Intel's math library (MKL) located in /soft/linux/pkg/intel/mkl70 if you application supports it.

Example of compilation (application appl consisting of appl1.f and appl2.c source code) which supports both MPICH and MKL:

setenv PATH /opt/mpich/myrinet/intel/bin:${PATH}
mpif77 -c appl1.f
mpicc -c appl2.c
mpif77 -o appl appl1.o appl2.o \
 -L/opt/mpich/myrinet/intel/lib -lmpichf90 -lmpich \
 -L/opt/gm/lib -lgm \
 -L/soft/linux/pkg/intel/mkl70/lib/32 -lvml -lmkl_lapack64 -lmkl -lguide -lpthread

Performance of the cluster

Here are some user contributed, real-world results of MD simulations on the cluster and comparison to other local resources.

1480 atom system, 10,000 steps, Amber

cluster	4p	8p	16p
ctbp1	288s
ctbp1m	169s	108s	86s

110,000 atom system, NAMD

ctbp1m	32p 0.7days/ns
a gigabit cluster w/ 3GHz Xeons	32p 0.98days/ns
datastar	128p 0.35days/ns

25,000 atom system, NAMD (constant pressure, particle mesh Ewald, and a cutoff of 10 A)

#procs	time/step	efficiency
2	0.26	100% (reference)
4	0.134	97.0
8	0.068	95.6
16	0.035	92.9
32	0.0186	87.4
64	0.0105	77.4

Courtesy of Donald Hamelberg and Justin Gullingsrud.