We have preliminary results on our “Utilizing 100G with a Single Server” project. Note that the results will be updated over the coming weeks.
The motivation for the this project is to develop a simple infrastructure capable of utilizing a 100G link using a single high performance server in Victoria and another in University of Michigan booth at SC2014. The data path goes via the Caltech booth at SC14 and forms part of the Caltech terabit network demonstration.
Each Dell R920 server system was able to drive the network memory at a perfectly flat 99.7 Gbps from UVic using a Brocade MLXe-4 to the Caltech booth onto the U Michigan booth SC14 show floor showing that we had a perfectly clean loss free network path between the two locations. The path was provided by CANARIE, BCNET and CenturyLink, and SCinet (the SC14 show network) in collaboration with Caltech Networking team.
With the systems configured as described in our initial project plan we were able to read from disk using SanDisk FusionIO drives over 65 millisecond latency between Victoria and the University of Michigan booth at 73 Gbps from disk and write at 60 Gbps, as shown in the figures below. These numbers show that we can have a single server handle the needs of a 100 Gigabit wide area circuit. All transfers were made using the open source FDT transfers software developed by Caltech. Extensive tuning of IRQs and server process binding with awareness of the Non-Uniform Memory Access architecture is required in order to achieve these levels of performance.
The infrastructure used to power these massive transfers at the University of Victoria is comparatively modest; only 9 rack units comprising a Brocade MLXe-4 switch Dell R920 server and 6 SanDisk FusionIO storage cards.
We believe that there is potential to improve on these results with further testing and tuning, however we are quite satisfied with the progress made during the limited time frame of SC14.
We need to further understand the interaction between all components of the long latency high bandwidth path with the multiple level of caching and buffering (Application, Driver, Filesystem, and Network) throughout the end-to-end system. We need to be able to maintain very smooth flow of data across the system end-to-end in order to be able to get the most from the hardware. It’s important to note that the network latency of 65 ms is 95% the result of the speed of light in an optical fibre between Victoria and New Orleans. In fact, locally the systems are able to write to disk at a shocking 113 Gbps and read at 137 Gbps using the commonly used FIO benchmarking tool.
Using a single 40GE network card we were able to move data disk-to-disk at 37.5 Gbps using just two SanDisk FusionIO drives. To us this indicates that we should expect to get to the final goal of 100G read at write given the extremely capable hardware.