During Super Computing 2011 a small team from Caltech, and University of Victoria transferred data from disks at the University of Victoria to disks at the show floor at more than 60 Gb/s (Gigabits per second). This is believed to be the fastest disk to disk Wide Area Network data transfer to be achieved at Super Computing 2011. In addition the team was able to move data from memory to memory at over 98 Gbit/s and receive it using only 4 machines on the show floor. Throughput of this magnitude would have required dozens or hundreds of high performance machines only a few years ago. The team was able to sustain indefinitely a total bidirectional memory to memory data transfer of 186 Gb/s. The demonstration showcased the performance possibilities of cutting edge 40 GE network cards combined with PCIe Generation 3 servers with solid state disks. During the course of the SC11 exhibition more then 4 Petabytes of data was moved between the two sites. Both Universities are Tier2 computing sites for the worlds largest scientific experiment the LHC in Geneva Switzerland. LHC experiments are seeing ever increasing demands for network bandwidth as data is being rapidly collected. University of Victoria Professor and LHC physicist Randall Sobie commented that:
” The 100Gb/s demo at SC11 is pushing the limits of network technology by showing that it is possible to transfer peta-scale particle physics data sample in a matter of hours to anywhere around the world. ”
Memory to Memory Transfers showing 4 servers receiving data at an aggregate speed greater than 98 Gb/s (top right corner). We also see a total bi-directional data flow of 186 Gb/s
The production-grade 100G circuit was constructed over the BCnet and CANARIE network, spanning 217 kilometers from the University of Victoria Computing Centre located in Victoria, British Columbia, to the Washington State Convention Centre in Seattle, Washington. The long haul optical transport was provided by Ciena OME 6500s. Each cluster was equipped with a Brocade MLXe-4 with 100 GE blade. The network hardware was generously contributed by our industrial partners and performed flawlessly throughout the exhibition.
This graph shows disk to disk transfers between servers at the University of Victoria and the Caltech booth peaking at over 60 Gb/s. The transfers were maintained for an eleven hour period on the 16th and 17th of November. Each color represents the data received by a single machine.
The University of Victoria employed a cluster of 10 Dell servers each containing six Solid State Disks (SSDs) and 10 Gigabit Ethernet interfaces. Each system was able to read from disk and deliver the data to the network at 9.4 Gb/s.
The Caltech booth employed a combination of PCIe Generation 3 and Generation 2 servers with 40 GbE Mellanox CX3 network cards. After careful tuning, single PCIe Gen 3 servers were able to receive an astonishing 35 Gb/s of memory to memory traffic and write to disk at 12.5 Gb/s (each machine containing 16 SSDs).
Data transfers were done with high performance open source data transfer tool FDT developed at Caltech.
Some of the team members at Super Computing 2011 posing in front of the network and storage throughput display at the Caltech Booth. The photo was taken shortly after SCinet WAN links were shutdown on the show floor November 17, 2011.
Full list of team members available here.
Media Releases From Involved Organizations
Contacts
Randall Sobie, Professor of Physics
rsobie@uvic.ca
+1 250 721 7733