• Home
• About Us
  News Archive Partners Contact Us Acknowledgments
• Systems
  maya Other systems
• Research
  List of Projects Publications Posters Presentation Handout Supporting Materials
• Resources for HPCF Users
  System description Using your account Tutorial: Compiling C programs Scheduling rules Tutorial: Running programs Other Languages and Hardwares Other Packages General Productivity Usage rules Account request form Help request

The cluster tara is an IBM Server x iDataPlex which was deployed in November 2009, with the release to HPCF users following in January 2010. Read on for a hardware-level description of this system. For more information about using your account on the system, see this page. The name tara originates from a Hindu goddess and means "star". tara is the second HPCF cluster named from Hindu scripture - the first was kali.

A few photos of the cluster

Nodes

tara has a total of 86 nodes which fall into four different categories. There is one front end node, one management node (for admins only), two development nodes, and 82 compute nodes. Each node features two (2) quad core Intel Nehalem X5550 processors (2.66 GHz, 8192 kB cache), 24 GB of memory, and a 120 GB local hard drive. All nodes run the standard UMBC distribution of the Linux operating system, Redhat Enterprise Linux 5. Attached to the cluster is 160 TB of central storage.

The Intel Nehalem processor series that is used in tara features some innovations that should be very beneficial to scientific computations. Specifically, each processor on a node has its own memory channels to its dedicated local memory, which should improve the speed of memory access. Other improvements by Intel aim at optimizing cache and loop performance and at improving head efficiency; for more information about Nehalem see Intel's website. Also Wikipedia provides a useful comparison between the various models of the chip that are available. Our own initial tests bear out the performance improvements of the processors, since they demonstrate that all cores on a node can be used simultaneously with profitable performance; see the Technical Report HPCF-2010-2.

More details on the different types of nodes in tara:

• Management node - There is one management node, which is reserved for administration of the cluster. It is not visible to users.
• Front End node - This is the node users of the cluster interact with directly. Users access all their files, edit code, compile jobs, and submit jobs to the scheduler from this node. It is reached through SSH by logging in to tara.rs.umbc.edu, and is the only node directly accessible from the outside network (that is, from outside of the cluster). Once logged into the front end node, it's hostname will appear as tara-fe1.rs.umbc.edu.
• Compute nodes - These nodes are where the majority of the computing will take place. You do not normally interact with these nodes directly. Instead, you submit jobs to the scheduler while logged onto the front end node. In turn, the scheduler will allocate your job to available processors on the compute nodes. The hostnames of the compute nodes are n3, ..., n84.
  In principle you could connect to them from the front end node (e.g. "ssh n84"). However, SSH access to the compute nodes is disabled to help maintain stability of the cluster.
  In general, any use of resources on the compute nodes should be allocated by the scheduler. Advanced users should be very careful when doing things such as spawning child processes or threads - you must ensure that you are only using the resources allocated to you by the scheduler.
• Development nodes - These are special nodes which are dedicated to running code that's in development. This allows users to test their programs, and not interfere with production runs. These nodes can be access through the 'develop' queue. Development jobs are expected to be small in scale, but rerun frequently as you work on your code.

  The availability of two development nodes allows you to try several useful configurations: single core, several cores on one processor, several cores on both processors of one machine, all cores on both machines, etc. The hostnames of the development nodes are n1 and n2.

Network Hardware

Two networks connect all components of the system:

• For communications among the processes of a parallel job, a high performance InfiniBand interconnect with low latency and wide bandwidth (quad data rate) connects all nodes as well as connects to the central storage. This amounts to having a parallel file system available during computations. Ideally the system can achieve a latency of 1.2usec to transfer a message between two nodes, and can support a bandwidth of up to 3.5 gigabytes per second. For more information about the network components, see:
  • Wikipedia article about Infiniband networks
  • Host Channel Adapter (HCA) Expansion Card
  • Switch
• A conventional Ethernet network connects all nodes and is used for operating system and other connections from the front end node to the compute nodes that do not require high performance. It also connects the long-term storage (see below) to the cluster.

Storage

There are a few special storage systems attached to the cluster, in addition to the standard Unix filesystem. Here we descibe the areas which are relevant to users of the cluster. See Using Your Account for more information about how to access the space.

• Home directory - Each user has a home directory on the /home partition. This partition is 200 GB and its data is backed up by DoIT. Since the partition is fairly small, users can only store 100 MB of data in their home directory.
• Scratch Space - All compute nodes have a local /scratch directory and it is generally about 100 GB in size. Files in /scratch exist only for the duration of the job that created them. This space is shared between all users, but your data is accessible only to you. This space is safer to use for jobs than the usual /tmp directory, since critical system processes also require use of /tmp
• Tmp Space - All nodes (including the front end node) have a local /tmp directory and it is generally 40 GB in size. Any files in a machines /tmp directory are only visible on that machine and the system deletes them once in a while. Furthermore, the space is shared between all users. It is preferred that users make use of Scratch Space over /tmp whenever possible.
• Central Storage - Users are also given space on this storage area which can be accessed from anywhere on the cluster. Spaces in central storage area are available for personal workspace as well as group workspace. Full backups are maintained for some of these spaces, while others have nightly snapshots taken which are kept for 10 days.
• UMBC AFS Storage on tara - Your AFS partition is the directory where your personal files are stored when you use the DoIT computer labs or the gl.umbc.edu login nodes. The UMBC-wide /afs can be accessed from tara.