The Canadian High Energy Physics network organization, HEPnet/Canada at the University of Victoria and CANARIE, together with their colleagues at Caltech, showed that researchers can fully exploit the new Advanced North Atlantic 100GE (ANA-100) 100 gigabit/second transatlantic link between NYC and Amsterdam provided by CANARIE and their partners Internet2 (USA), NORDUnet, (Nordic countries), ESnet (U.S. Department of Energy), SURFnet (Netherlands).
High performance networking testbeds at CERN in Geneva Switzerland and Ottawa, Canada were connected with the ANA-100 transatlantic circuit and a series of increasing bandwidth memory-to-memory network transfer were performed. The team was able to reach a maximum sustained network throughput from CERN to Ottawa of 94 Gbps using only two specially configured Linux servers in each location. In addition a peak bidirectional average throughput of 127 Gbps was sustained over a period of 11 hours.
ANA-100 traffic while moving data from CERN to Ottawa on May 5, 2014. The rate peaks at a sustained 94 Gbps.
Each server used in the demonstration was configured with two Mellanox 40GE Network cards. The transfers were performed using only 3 pairs of cards and 25 parallel TCP streams in total. All transfers were completed using the high performance, open source, data transfer tool FDT developed by Caltech and others.
Optical transport across the ANA-100 link and within Canada is done using Ciena Equipment. The 100G transatlantic demo was enabled by Juniper Networks, who loaned the group a 100G router, and by CERN, who gave us access to their networks and computing facilities. Scalar Decisions acted as supplier for the servers used in Ottawa.
ANA-100 CANARIE Testbed located in Ottawa Canada.
During the Super Computing 2013 (SC13) conference November 18-21, an international team of high energy physicists, computer scientists, and network engineers from the California Institute of Technology (Caltech), the University of Victoria (UVic), and the University of Michigan , Vanderbilt , University of São Paulo, and the laboratories of KIT, CERN, LBNL and BNL performed a number of different demonstrations using a variety of next generation network equipment and servers.
Figure 1: Intercontinental OpenFlow Testbed formed for SC13. Links Victoria – Denver- São Paulo – Chicago – New York – Amsterdam – CERN (in Geneva) .
This year UVic with its partners BCNet and CANARIE focused on a demonstration of Intercontinental OpenFlow tested spanning Victoria – Denver – San Paulo – Chicago – New York – Amsterdam – CERN (in Geneva) as shown in Figure 1. The Western most node of the Canadian Software Defined Network testbed at UVic was connected with a 10G link leveraging the Internet 2 Advanced Layer 2 Services (AL2S) switch in Seattle. Two diverse paths were carried over Internet 2 terminating on two separate switches on the Super Computing show floor. Physics data was transferred over this network using OpenFlow link layer multipath switching to efficiently utilize the multiple network paths to form a loop free topology. The multipath utilization is completely transparent to the end hosts of the network and accomplished using the Caltech developed OLiMPS OpenFlow controller that resided at CERN in Switzerland. This methodology allows an automated in-network load balancing of various flows to increase network utilization and efficiency. Moreover, multiple network links can be arbitrarily added to a network to increase capacity and redundancy without the traditional challenges of Layer 2 link aggregation using LACP.
During the demonstration we were able to establish perfectly balanced network traffic across the multiple links of the testbed in a configuration that would normally cause a network loop. Figure 2 shows the results of transferring files from a single machine located on the show floor in Denver.
Figure 2: perfect balancing across diverse layer 2 paths between Victoria and Denver.