HPC User Site Census: Interconnects/Networks

by Addison Snell, Christopher G. Willard, Ph.D., Laura Segervall
for Intersect360 Research
January 2010

This third report in our 2009 Site Census series provides an examination of the server interconnect and network characteristics found at a sample of HPC user sites. We surveyed a broad range of users about their current computer system installations, storage systems, networks, middleware, and software supporting these computer installations. The other completed reports in this series include: HPC User Site Census: Systems and HPC User Site Census: Processors.

Our goal in this analysis of computer system interconnect and network implementation is to examine the suppliers, type, and purpose of the network within the HPC user communities, and to explore how this usage varies between computer systems interconnects, storage networks, and local area networks.

Key findings of the survey include the following:

• Ethernet and Infiniband are the only two standards that compete across all three locales (system interconnects, storage networks, and LANs). Infiniband has the greatest representation for system interconnect with 39%, with the percentage increasing significantly year by year. Ethernet has the greatest representation for storage networks (65%) and local-area networks (87%).
• Use of Ethernet and Infiniband at the other locales appears to be dependent on its heritage. That is, use of Ethernet for a system interconnect or storage network at a site is more likely if the site has an Ethernet LAN and the use of Infiniband for storage networks and LAN is more likely if it is being used as a system interconnect at that site.
• Cisco was the supplier used most often for all three locales with 16% mentions for system interconncets, 31% for storage networks, and 53% for LANs. System interconnect is the only locale where Cisco is being challenged. Myricom is close behind Cisco with 15% of the mention.
• Most NAS storage systems (63%) are connected to 1 Gigabit Ethernet networks. SAN storage systems are evenly divided with 39% connected to a Fibre Channel network and 37% connected to an Ethernet network. Infiniband accounted for 18% of the SAN storage networks.
• Ethernet, as expected, is the dominant LAN network implemented at these sites with 87% of all the networks being some type of Ethernet (some sites had more than one type installed). Approximately 94% of the sites had an Ethernet LAN network installed. 1 Gigabit Ethernet is still the primary type with 48% of the mentions. Cisco is the primary supplier for Ethernet and Infiniband implementations.

Sign in to download the full report