Preliminary results from SC14

Tags:

We have preliminary results on our “Utilizing 100G with a Single Server” project.  Note that the results will be updated over the coming weeks.

Network between UVic and the SC show floor

Network between UVic and the SC show floor

The motivation for the this project is to develop a simple infrastructure capable of utilizing  a 100G link using a single high performance server in Victoria and another in University of Michigan booth at SC2014. The data path goes via the Caltech booth at SC14 and forms part of the Caltech terabit network demonstration.

Traffic flow memory-tomemory from a Single R920 server located at the University of Victoria the UMichigan Booth on the SC14 Show floor.

Traffic flow memory-tomemory from a Single R920 server located at the University of Victoria the UMichigan Booth on the SC14 Show floor.

Each Dell R920 server system was able to drive the network memory at a perfectly flat 99.7 Gbps from UVic using a Brocade MLXe-4 to the Caltech booth onto the U Michigan booth SC14 show floor showing that we had a perfectly clean loss free network path between the two locations. The path was provided by CANARIE, BCNET and CenturyLink, and SCinet (the SC14 show network) in collaboration with Caltech Networking team.

With the systems configured as described in our initial project plan we were able to read from disk using SanDisk FusionIO drives over 65 millisecond latency between Victoria and the University of Michigan booth at 73 Gbps from disk and write at 60 Gbps, as shown in the figures below. These numbers show that we can have a single server handle the needs of a 100 Gigabit wide area circuit. All transfers were made using the open source FDT transfers software developed by Caltech. Extensive tuning of IRQs and server process binding with awareness of the Non-Uniform Memory Access architecture is required in order to achieve these levels of performance.

Reading from SanDisk FusionIO disk at the University of Victoria at writing to memory at University of Michigan Booth. We see a peak of 73 Gbps. Each color shows the thoughput from a single 40GE network card and the total throughput is shown in the top curve.

Reading from SanDisk FusionIO disk at the University of Victoria at writing to memory at University of Michigan Booth. We see a peak of 73 Gbps. Each color shows the thoughput from a single 40GE network card and the total throughput is shown in the top curve.

 

Disk write over the wide area network from UVic to the UMichigan Booth peaking at 60 Gbps. The top curve shows the sum of thoughput from 3 x 40GE interfaces. The lower cures show the thoughput from the individual network interfaces.

Disk write over the wide area network from UVic to the UMichigan Booth peaking at 60 Gbps. The top curve shows the sum of thoughput from 3 x 40GE interfaces. The lower cures show the thoughput from the individual network interfaces.

The infrastructure used to power these massive transfers at the University of Victoria is comparatively modest; only 9 rack units comprising a Brocade MLXe-4 switch Dell R920 server and 6 SanDisk FusionIO storage cards.

Total resources at UVic for the 100G server project: Brocade MLXe switch, Dell R920 with 6 SanDisk FusionIO  cards.

Total resources at UVic for the 100G server project: Brocade MLXe switch, Dell R920 with 6 SanDisk FusionIO cards.

We believe that there is potential to improve on these results with further testing and tuning, however we are quite satisfied with the progress made during the limited time frame of SC14.

We need to further understand the interaction between all components of the long latency high bandwidth path with the multiple level of caching and buffering (Application, Driver, Filesystem, and Network) throughout the end-to-end system. We need to be able to maintain very smooth flow of data across the system end-to-end in order to be able to get the most from the hardware. It’s important to note that the network latency of 65 ms is 95% the result of the speed of light in an optical fibre between Victoria and New Orleans. In fact, locally the systems are able to write to disk at a shocking 113 Gbps and read at 137 Gbps using the commonly used FIO benchmarking tool.

Using a single 40GE network card we were able to move data disk-to-disk at 37.5 Gbps using just two SanDisk FusionIO drives. To us this indicates that we should expect to get to the final goal of 100G read at write given the extremely capable hardware.

 

 

 

 

 

100G link SC 2014 Victoria-New Orleans fully utilized

The network between Victoria and the University of Michigan booth at SC 2014 in New Orleans is running very well.   The system was left running over night (Nov 18-19) with data being transferred from New Orleans to Victoria at the full 100G capacity of the link.

The red plot is the traffic into UVIC and the yellow plot is the traffic from the UMichigan booth.

Network-since-0200