Cosmological simulations have found >100 times more substructure than observed, so astronomers want to understand the discrepancies. Researchers in Tokyo and Amsterdam are building an intercontinental supercomputer grid to run cosmological N-body simulations of 10 billion particles.
Simon Portegies Zwart of UvA and Junichiro Makino of NAOJ proposed the CosmoGrid experiment, to model the detailed dark fine structure in the Universe with 10 billion dark-matter particles using two supercomputers concurrently. Half the universe is calculated on the Amsterdam supercomputer “Huygens” located at the SARA supercomputer center, and the other half is calculated on aCray XT4 at the Center for Computational Astrophysics in Tokyo. The two computers are connected using optical networks between Amsterdam and Tokyo. Currently, the team has successfully run more than 100 time steps of the planned 50,000. The entire calculation is expected to take about a year, within which they plan on spending about 3million hours of CPU time.
JGN2plus requested use of the TransLight/StarLight CHI/AMS circuit to support this. On May 29, 2008, a direct connection between NAOJ and UvA was created. A test run took place in June 2008; a full simulation took place later in the year.
Given preliminary results between Tokyo and Amsterdam, on June 29, Zwart submitted the grant proposal “The Gravitational Billion Body Problem” to the DEISA (Distributed European Infrastructure for Supercomputing Applications) Extreme Computing Initiative. The goal is to do high-performance grid computing by connecting five supercomputers to perform a large-scale cosmological simulation, similar to this CosmoGrid (Amsterdam-Tokyo) project, but on a cosmological scale. The Organization for the Netherlands Supercomputer Facilities (NCF) granted CosmoGrid funding, and forwarded it to DEISA as the #1 proposal from the Netherlands. On October 25, the principals received word that DEISA awarded them 3,150,000 computer hours from January 1 to December 31, 2009.
On October 17, the team reported that its first full test simulation using TreePM code and the SocketLibrary concurrently on two supercomputers – the Cray XT4 in Tokyo and the Huygens in Amsterdam – was successful. One full simulation from z=60 was completed using more than 17million (256^3) particles. The first results, which include an animation, are available on the project’s wiki.
Immediately following SC08, the network engineers established a lightpath between NAOJ in Japan and SARA in The Netherlands for further network testing January 15-31, 2009. The scientists successfully ran a 2k^3 simulation concurrently on the Cray in Tokyo and the Huygens supercomputer in Amsterdam. StarLight’s MRTG traffic utilization graph showed a sustained flow of 30-40Mbps, but the traffic was bursty (with peaks over 500Mbps). Note in the diagram below that the Tokyo to StarLight path went over the IEEAF link to Seattle, and then over the TransLight Cisco Research Wave to Chicago.
URL:
http://modesta.science.uva.nl/Projects/2008/CosmoGrid/
http://wiki.2048x2048x2048.org/
Collaborators:
USA:
Drexel University; Vanderbilt University; StarLight
Canada:
CANARIE
Japan:
University of Tokyo; National Astronomical Observatory of Japan (NAOJ)/Center for Computational Astrophysics; JGN2plus; SINET3; T-LEX
Netherlands:
University of Amsterdam (UvA); SARA; SURFnet
UK:
University of Edinburgh/School of Mathematics
Germany:
Ludwig-Maximilians Universität at München
