At Colorado State University we recently had the good fortune of upgrading an aging CrayXT6m high-performance computer to a newer model XE6. With an additional 672 CPU cores, faster Interlagos processors, and 4X faster Gemini interconnect, the upgrade approximately doubles the performance of the system. With a large installed software base, our user community will see a nice boost in many of our parallelized applications. In the preceding five years, the XT6m provided a highly stable, general purpose computing platform for parallelized applications. The XE6 promises the same.
After the upgrade the basic Cray specs are:
- 2,688 CPU cores; higher levels of code parallelization and increased job throughput
- 2.5 TB aggregate RAM
- 32 TB disk space (expansion coming soon)
- Gemini 3D-torus interconnect; enables faster MPI message passing, all-important for MPI-enabled parallel code
- 168 16-core AMD Interlagos processors; faster CPU performance
- 10gig NIC’s; faster file transfers
- Large installed software application base
Arguably, because I like to argue, the most important component of the upgrade may be the Gemini interconnect. The interconnect, like our phone system infrastructure, is largely invisible unless you’re a software developer writing parallelized code. In the background, Gemini links Interlagos processors (compute nodes) in a 3-D torus design and routes MPI messages between nodes. Most of the software applications installed on the Cray are MPI-enabled, and thus use MPI message passing for parallelism, scaling, and performance. The speed of the interconnect directly correlates to the speed of MPI message passing. The Gemini interconnect is 4X faster than the older Seastar2+ interconnect. Hence, we expect significant improvements in performance simply due to the increased speed of Gemini. Extra CPU cores are nice, but Gemini may be the star in this upgrade.
Although we tend to focus on the hardware aspects of upgrades like this, it’s the software applications that are a real challenge. Yes – we’ll have to reinstall ca. 200 applications to bring the system back up to its previous state! But that’s where software package managers shine. With the help of SPM’s, great tech support from Cray Inc. and the work of some talented graduate students, everything should be humming again in a few days.
In an upcoming blogpost I’ll focus on one aspect of the upgrade, namely, the integration of Next Generation Sequencing datasets, Bioinformatics software applications, and the Cray high-performance computing environment. Stay tuned.