Calculable black holes
Cluster computers based on Intel Xeon architecture at the Albert Einstein Institute are now twice as fast
Almost 100 years after Albert Einstein developed the theory of relativity, our knowledge of gravitational physics has grown rapidly. Einstein, for example, had strong reservations about the existence of black holes. Today we are certain that they exist. This certainty regarding black holes is due, inter alia, to advances in computer research. Ever faster computers today enable scientists to test complex concepts and theories of gravitational physicists by simulating realistic astrophysical systems and visualizing them three-dimensionally.
At the Max Planck Institute for Gravitational Physics (Albert Einstein Institute, AEI) in Potsdam-Golm, an international centre of relativity, some of the leading minds of gravitational physics are to be found.
For the highly complex numerical simulations of black holes, ever greater demands are placed on computing power, storage capacity, and the network speed of the computers used. For this type of computation the AEI uses the PEYOTE cluster computer, which was doubled to 128 computer nodes at the end of 2003, thereby allowing users to further refine their model calculations. The support of INTEL made it possible to test the latest processor of the company, a P4 Xeon with 3.06GHz and 1MB 3rd Level Cache. This was undertaken before the official launch of the product, so that one could balance possible increases in performance by using this type of processor against the additional costs. The latest developments in the design of the expansion could also be taken into account thereby. This year a further expansion to 256 PC nodes is planned for the cluster; later an increase to 1024 processors is planned.
PEYOTE
The main users of the cluster are the scientists in the Department of Astrophysical Relativity and, in particular, the Numerical Relativity Group. However, scientists from other groups and cooperating partners are also users of this high-performance cluster. Especially close ties exist with Prof. Dr. Ed Seidel, the Director of the Center for Computation and Technology at Louisiana State University (LSU) in Baton Rouge and former head of the Numerical Relativity Group at the AEI. Prof. Seidel, who still continues to work at AEI in combination with his position at LSU, emphasizes: "This collaboration enables our groups to continue at the vanguard of research on black hole physics. By pooling resources at AEI and at Louisiana State University, we are also creating a powerful grid environment for collaborative research."
The cluster is particularly suitable for problems that can be parallelized. These are matrix operations, as they are also primarily used for simulations. In such instances, the individual nodes of the cluster must be able to communicate with one another with particular speed and effectiveness. The main research area of the Numerical Relativity Group is the calculation of the Einstein equations for astrophysically interesting cases such as the collision of black holes or neutron stars. The results of these simulations are visualised either on the head node of the cluster or on workstations that are particularly suitable for graphics output.
The Max Planck Institute for Gravitational Physics
The Max Planck Institute for Gravitational Physics (Albert Einstein Institute) in Potsdam-Golm has established itself as a leading international research centre for gravitational physics since it was founded in 1995. Over 90 scientists and more than 150 visitors each year investigate all aspects of gravitational physics.
The cluster / technical data
A high-performance Linux Compute Cluster is being used. After extension, the cluster consists of 128 computing nodes, 64 of which have 2 Intel XEON P4 processors each (2.66 GHz and 2 GB of RAM). 64 additional nodes also have 2GB RAM, but are equipped with 2 Intel XEON processors with 3.06 GHz clocking. Both types have a 120GB hard drive as local storage space. The four file servers, each with 1TB storage space, have been increased by another four. At the same time the existing ones were expanded, so that now 8 x 1.5 TB is available (1 terabyte = 1,000 gigabytes). For comparison: an ordinary PC has a hard drive capacity of 80-120 GB.
The core of the high-performance cluster is the network and thereby the corresponding switch that provides for inter-process communication. This switch was provided by the Force10Networks company. In this regard, special importance is placed on short latency periods and delay-free data transmission, both of which are guaranteed by GigaBit Ethernet technology.
Although today Myrinet, a high-performance switching technology, often plays an important role, the choice was made in favour of GigaBit, because it is more or less the standard and therefore also promises favourable extension possibilities in the future and was found to have an optimal price/performance ratio.
Because typical computer runs take several days or even weeks, the runs are administered by a batch system. The users employ management nodes to communicate with the cluster. It is there that programmes are compiled and the results are visualised on nodes. An extremely important part of all computational tasks of scientists at AEI is played by the CACTUS Code (www.cactuscode.org), developed by AEI. It is a flexible selection of tools and makes it easy for all scientists involved to formulate problems in a computer-friendly way and to have calculations carried out.
Technical data
64 PC nodes with 2 processors @ 2.66 GHz/533 FSB (front side bus)
64 PC nodes with 2 processors 3.06 GHz/533 FSB (front side bus). Processor type: Intel XEON with hyper-threading technology.
Per node:
2 GByte RAM memory
120 GB storage capacity
3 network cards
8 storage nodes, each with 1.5 TB storage capacity
2 head nodes (also called access and management nodes)
Storage and head nodes are similar to computer nodes, but have 4GB of RAM memory and do not need an interprocess network. The system board and power supply units are designed redundantly.
The operating system Linux with RedHat distribution is installed on all computers.
Details
Each computer node has three network cards for three specific networks. The most important is the InterconnectNetwork which, by using computer nodes with 1,000 Mbits (1Gbit), connects lines by means of a very powerful switch. A Force10Networks company switch is used (for a switch description see below). It has a back plane (BUS circuit board) capacity of 600 Mbits.
The second network, which is also very important, is used to transfer the results of individual nodes to the so-called storage nodes. Because of the enormous data output of the computer nodes, the best approach is to distribute the load over several nodes. For example, the output of 16 nodes is written to one storage node. The network uses an HP ProCurve 4108gl switch. It has a back plane of 36 Gbits. This is sufficient to deal with a load of 4 Mbytes from each [!] computer node at the same time.
The third network ensures that all components of the cluster can be operated. For this purpose a switch from the HP company has been used. In order to keep the cable length as short as possible, two further switches are used in this network. The manufacturer of these is the 3Com company.
Cooling of the cluster
Because of the small space required for the equipment, SlashTwo housing was chosen. This packed form requires special consideration of air flows in the housing, since the processors give off an enormous amount of heat that has to be transported away as quickly as possible. The temperature of the ambient air should not exceed 20°C. It is necessary to ensure that an air volume of 4 x 1400 m2 is available and can be recirculated. The existing air conditioning system can handle these values and has an output of ca. 50kW. The existing ceiling units are available as a reserve for especially sunny days in the summer months and provide an additional 24 kW.
Power supply
The cluster is supplied with 6x25A lines. A UPS (uninterrupted power supply) ensures an even power supply of the storage and head nodes for a period of 20 minutes. Special software then ensures that these computers automatically shut down and are turned off.
Further specifications
Cluster
Pro Rack (19" cabinet):
Weight: 400 kg with 16 SlashTwo housing
240 kg for the network cabinet
250 kg for the rack with the storage nodes and head nodes, including the 6 TB storage units
Network
Force10Networks E600
Weight: 110 kg
Power consumption: 2800 W
Waste heat: 1400W - 3500W
Network specifications
Back plane capacity: 600 Gbits
Special software
Although the cluster can be regarded as one unit, the individual components must nonetheless be capable of individual use as software.
In this regard special software, so-called management software, simplified matters enormously. For this purpose the Megware company has developed a cluster management software by the name of Clustware.
Special features of the Cluster
Peak performance of the Cluster
Theoretically:
1. Cluster half:
128 x 2 x 2,66GHz = 680GFlops (128 CPUs, 2 floating point units per CPU, 2.66GHz per unit)
2. Cluster half:
128 v 2 3.06GHz = 783GFlops (128 CPUs, 2 floating point units per CPU, 3.06GHz per unit)
Total: 1.46 TFlops
The true values will be reflected in the benchmarks.
A single PC, when used in the Cluster (2 CPUs) and available as a desktop workstation for individual scientists at the AEI, has a performance level of 10 Gflops.
The communication network is based on Gigabit Ethernet. This network card was chosen, based on the assumption that the development of Ethernet would continue.
The interprocess switch is already designed for 10 Gigabit Ethernet.
