1. Introduction

With the emergence of cloud computing and high-bitrate video services, data center applications have attracted much attention of service providers and network operators. An increasing number of service providers and enterprises are hosting their storage contents and computing resources in data centers to achieve lower delay and higher availability at a lower cost. Due to the diversity and hugeness of the services, the high-performance network-based data center applications have presented the high burstiness and high-bandwidth characteristics, especially for the super-wavelength application beyond 100Gbps. The traditional wavelength-division-multiplexing (WDM) optical transport networks are inefficient and unsuited for carrying these applications due to the strictly fixed ITU-T wavelength grids and spacing. To meet the flexible bandwidth demand, the architecture of elastic optical networks has been proposed and experimentally demonstrated [1], which can be achieved by taking advantage of orthogonal frequency division multiplexing (OFDM) technology and bandwidth-variable wavelength cross-connects (BV-OXCs) [2]. The key idea of such networks is to allocate the necessary spectral resources with a finer granularity tailored for a variety of user connection demands. The architecture of elastic optical networks can also enable sub-wavelength, super-wavelength, multiple-rate data traffic accommodation, and offers cost-effective, highly-available and energy-effective connectivity channels. In order to support the service operation flexibly, data center interconnection by elastic optical networks is a promising scenario to allocate spectral resources for applications in a dynamic, tunable and efficient control manner.

Additionally, a large amount of network-based data center applications require the end-to-end guaranteed quality of service (QoS). Depending on the technological heterogeneity and resource diversity, the services delivery guaranteeing end-to-end QoS is practically impossible in independent operation scenario with cross stratum optimization (CSO) [3, 4], which allows global optimization and control across optical network and data center resources [5]. Recently, as a centralized software control architecture, the software defined networking (SDN) enabled by OpenFlow protocol has gained popularity by supporting programmability of network functionalities and protocols [6–8], which can provide maximum flexibility for the operators and make a unified control over various resources for the joint optimization of functions and services with a global view [9, 10]. Therefore, nowadays operators are trying to apply SDN/OpenFlow techniques to globally control network and application resources in such data center interconnection by elastic optical networks [11].

In this paper, we propose and implement a novel elastic optical network based on enhanced software defined networking (eSDN) architecture for data center application, by introducing a transport-aware cross stratum optimization (TA-CSO) strategy. eSDN can enable cross stratum optimization of application and elastic optical network stratum resources, and can provide the elastic physical layer parameter adjustment (e.g., modulation format and bandwidth) based on the distance of path and network resource. The overall feasibility and efficiency of the proposed architecture is also experimentally demonstrated on our testbed with four real OpenFlow-enabled elastic optical node devices in terms of path setup/release/adjustment latency, blocking probability and resource occupation rate.

The rest of this paper is organized as follows. Section 2 proposes the novel elastic optical network based on eSDN architecture and builds functional models and protocol extension for eSDN. The interworking procedure for eSDN under this architecture is described in section 3. Section 4 shows the experimental demonstration and performance evaluations of our testbed. Section 5 concludes the whole paper by summarizing our contribution and discussing our future work on this area.

2. Elastic optical networks based on eSDN architecture for data center application

The use of SDN/OpenFlow in packet-switched and circuit-switched networks has been widely studied in protocol as well as device. Most studies focus on the features of OpenFlow enabling networks technologies, including Flexi-Grid and Fixed-Grid optical networks [8]. However, the issue about the data center application resources and QoS requirement of users has not been addressed in this elastic optical networks scenario. With the increment of data center application and network elasticity, the scheduling of data center resources and elastic network resources can inevitably bring pressure to the QoS guarantee of users. The traditional OpenFlow centralized architecture based on the single stratum (i.e., network stratum or application stratum) cannot support and meet the scalability and flexibility requirements for SDN. In this section, we focus on the elastic optical networks based on eSDN architecture for data center application. The architecture focuses on two stratum resources, i.e., data center application and network resources. Each stratum resource is software defined with OpenFlow and controlled by application controller (AC) and transport controller (TC) respectively in a unified manner. Compared to traditional OpenFlow centralized controller, the eSDN controllers include not only the flow table, path computation and corresponding modules, but application service management, TA-CSO strategy and application and network information maintenance and so on. Through the interworking between application controller and transport controller, the eSDN architecture can enhance the cross stratum optimization of transport network and data center application resources. First, the main core and structure of the novel architecture is briefly pointed out. After that, the functional building blocks of eSDN and coupling relationship between them as well as the main relative protocol extensions and strategy are presented in detail.

2.1 Elastic optical networks based on eSDN architecture

The eSDN architecture over elastic optical networks for data center application is illustrated in Fig. 1. The distributed data center (DC) networks are interconnected with elastic optical networks, which deployed application (e.g., CPU and memory) and network stratum resources respectively. Each stratum resource is software defined with OpenFlow and controlled by application controller and transport controller respectively in a unified manner. To control the heterogeneous networks for data center application with extended OpenFlow protocol (OFP), OpenFlow-enabled elastic optical device nodes with OFP agent software are required, which are referred to as software defined OTN (SD-OTN). The motivations for the eSDN architecture over elastic optical networks are twofold. Firstly, eSDN can emphasize the cooperation between AC and TC to realize CSO with joint and global interworking of cross stratum resources. Secondly, based on the distance of path and network resource, elastic optical networks can adjust the physical layer parameter (e.g., bandwidth and modulation format) with eSDN controlling to realize the software defined path (SDP) and optimize resource utilization. Based on functional architecture described above, a TA-CSO strategy is proposed in the AC to realize the application and network stratum resources optimization considering the tunable bandwidth and modulation format parameters.

 

Fig. 1 The architecture of elastic optical networks based on eSDN.

Download Full Size | PDF

2.2 Functional models of eSDN over elastic optical networks

The functional building blocks of application and transport controllers and the coupling relationship between different modules are shown in Fig. 2. The basic responsibilities and interactions of AC and TC are described as follows. The responsibility for the AC is concerned with monitoring and maintaining application stratum resources in data center servers for eSDN, while TC sustains optical network stratum information abstracted from physical network and lightpath provisioning in elastic optical networks. In TC, two control modules, i.e., physical network control and abstraction network control are included. The physical network control module is responsible for discovering physical layer network elements and controlling the spectrum bandwidth and modulation format in underlying layer network. The abstraction network control module can abstract and manage the network topology through path computation element (PCE) computation, which is capable of computing a network path or route based on a network graph, and of applying computational constraints [12, 13]. The abstraction network control module can also provide abstracted resource information to AC. The AC is responsible for the application service management and implements TA-CSO computation with both application and abstracted network resource stored in internal database. Once a data center service request arrives, application service management in AC can map it into request parameters and forward it to TA-CSO module which decides to implement TA-CSO strategy of application and network resource information. After completing the CSO computation, AC chooses the most optimal server or virtual machine for users, allocates application and spectral resources considering the tunable modulation format parameters and determines the location of application or where to migrate virtual machines. According to the result, AC transmits application requirements to TC via application-transport interface (ATI). On receiving service request from AC through ATI, the end-to-end SDP can be calculated and provisioned by controlling all corresponding SD-OTNs along the computed path by using extended OFP in TC. The application-transport interface (ATI) is an interface between application controller and transport controller, which carries the request and reply message and the information between application stratum and transport stratum. Note that, the modulation format of service (e.g., QPSK and 16QAM) is determined and adjusted based on the length of SDP. If the SDP has short distance, more precious spectrum bandwidth can be economized due to the use of high-level modulation format.

 

Fig. 2 The functional models of application and transport controllers (AC and TC).

Download Full Size | PDF

The functional models of SD-OTN including software and hardware are described in Fig. 3(a). In SD-OTN, OpenFlow-enabled agent software is embedded to keep the communication between TC and optical node, which achieves the OpenFlow protocol process, controls over modeled node and mapping to the physical hardware. Through the agent software, the SD-OTN can maintain optical flow table and modeled node information as software and map the content to configure and control the physical hardware, and receive the report of spectrum control. The hardware of SD-OTN is composed of a series of physical boards (e.g., flexible ROADM board, flexible ODU board and corresponding tributary card) due to the expansibility and convenience. The flexible ROADM and ODU boards meet the non-blocking and gridless requirement, and support various modulation format of the signal (i.e., BPSK, QPSK, 8PSK, 16QAM and 64QAM), as shown in Figs. 3(b)-3(c) respectively.

 

Fig. 3 The functional models of (a) OpenFlow-enabled SD-OTN, (b) Flex ROADM and (c) Flex ODU.

Download Full Size | PDF

2.3 Transport-aware cross stratum optimization (TA-CSO) strategy

The cross stratum optimization of data center and elastic optical networks architecture is represented as G (V, L, F, A), where V denotes the set of bandwidth-variable optical switching nodes, L indicates the set of bi-directional fiber links between nodes in V. F is the set of spectrum sub-carriers on each fiber link and A denotes the set of data center servers, while N, L, F and A represent the number of network nodes, links, the spectrum sub-carriers and data center nodes respectively. In each data center server, two time-varying application stratum parameters describe the service condition of data center application resources, which are comprised of memory utilization $U_R^{(t)}$ modeled RAM and CPU usage$U_C^{(t)}$. From another perspective, the parameters in network stratum contain the hop $H_p$of each candidate path, and the occupied spectrum bandwidth $B_l$ and distance $D_l$of each link, which are related to traffic cost of the corresponding link and the choice of modulation formats. From the users’ point of view, they pay much attention to the experience of QoS rather than concerning about which server to provide services. Therefore, for each application request from source node s, it can be translated into the needed network and application resources. Note that these resources contain the required network bandwidth b and application resources ar in the analysis of network model for simplicity. We denote the ith traffic request described above as $T{R}_i(s,b,ar)$, while TR_{i + 1} will arrive after connection demand TR_i in the time order. Additionally, according to the traffic request and status of resources, the appropriate data center server can be chosen as the destination node based on the strategy.

Based on the functional architecture of elastic optical networks based on eSDN architecture, we propose a novel TA-CSO strategy for the optimization between application and network stratum resources considering transport parameters. With TA-CSO strategy, AC selects the server node and data center location as the destination based on the application status collected from data center networks and network condition provided by TC dynamically. Receiving a new traffic request, AC verifies this demand and maps it into request parameters, i.e., $T{R}_i(s,b,ar)$. Due to the time changing characteristic of parameters, the adjustable evaluation rank rate $k_C,k_R$ among CPU and RAM utilization are used to describe the relative proportion of them. In order to facilitate the realization of TA-CSO strategy on the real testbed (mentioned later in the paper), the settings of the evaluation rank rate in strategy can be simplified appropriately when the simplification does not impact the process and effects of this strategy. We make the continuous rank value discretized and assume several typical values for simplicity, i.e., ${\text{R}}_a,{\text{R}}_b$. Initially, the evaluation rank of CPU is higher than RAM. At this point, evaluation ranks satisfy the expressions as follows: $k_C={\text{R}}_a\text{,}{k}_R={\text{R}}_b\text{,}$ ${\text{R}}_a+{\text{R}}_b=1,{\text{R}}_a\ge {\text{R}}_b$. ${\text{R}}_a,{\text{R}}_b$ are constants and their priorities decrease gradually. That means the higher usage corresponds to higher priority. Once the statistical average of RAM utilization exceeds CPU, the evaluation rank of them will be adjusted according to this change as follows: $k_C={\text{R}}_b,k_R={\text{R}}_a$. By parity of reasoning, $k_C,k_R$will be modified dynamically based on the feedback of statistical average variation. Therefore, the application occupation f_ac with the application stratum parameters of current each server is expressed as dimensionless function (1), where these parameters are normalized to meet the linear relationship between them.

In addition, following the method mentioned above, adjustable evaluation rank rate $k_B,k_D$between bandwidth and distance can be dynamically modified in network stratum. So, the network occupation f_bc with parameters of current each node is expressed as dimensionless function (2). In this equation, the parameter B and B_l denote the total bandwidth and occupied bandwidth of the link respectively.

Among the data center nodes, the candidate servers with the first K minimum of application functions are chosen by AC and expressed as the set F_a = $\left\{f_{a1},f_{a2},\mathrm{...},f_{ak}\right\}$. Then, the candidate path between source and each candidate server can be calculated with minimum network function and denoted as F_b = $\left\{f_{b1},f_{b2},\mathrm{...},f_{bk}\right\}$. From the view of vector graphics, F_a can be also seen as an K-elements-size vector space of K application occupation vectors $f_{a1},f_{a2},\mathrm{...},f_{ak}$. The mean vector $\overline{f_a}$ of vector space F_a expresses the center of them. The distance between the vector $f_a$ and the mean vector $\overline{f_a}$ is illustrated by ${\Vert f_a-{\overline{f}}_a\Vert }_2$. Among these vectors, the vector $f_{ai}$ and $f_{aj}$ are the farthest and nearest to the mean vector $\overline{f_a}$, which are chosen by Eq. (3). The correlation coefficient of the vector $f_{ai}$ and $f_{aj}$ is calculated as $\beta$, which is shown in Eq. (4). The correlation coefficient is related to the degree of data center load balancing, which means the variance among application occupations in data centers. The larger coefficient represents that the balancing degree becomes better in servers. The reason is the correlation coefficient of application occupation on different servers represents the correlation degree of them. The larger value of correlation coefficient leads to the greater interdependence of the servers’ load, so as to vary with each other. Therefore, the larger coefficient denotes the load of the servers can be more balanced and balancing degree becomes better in servers. We define $\alpha$as the joint optimization factor to assess the resource utilization globally in application and network stratums, while the dynamic weight between the network and application parameter is described as$\beta$. Based on the formulas described below (4), the application utilization weight $\beta$ changes dynamically according to the feedback of load balancing degree. So the joint optimization factor $\alpha$meets the formula as follows (5).

According to application and network utilization, the node with minimum$\alpha$value based on the joint optimization factor will be selected from the K candidates as the destination node. Receiving the traffic request and pairs of source and destination node from AC, TC can complete the end-to-end path computation in the connection and service parameters constraints, and then select the appropriate modulation format according to the SDP distance and perform spectrum assignment for the computed path and the lightpath provisioning by OpenFlow protocol.

2.4 Extension of OpenFlow protocol for eSDN over elastic optical networks

In the packet-switched network, OpenFlow abstracts data plane as a flow entry which is defined as rule, action and statistics [10]. It represents the packet’s characteristic (e.g., IP address and TCP/UDP port) and the action of switch. For the eSDN control of elastic optical networks, flow entry of OpenFlow protocol in optical flow table (Fig. 3) is extended and illustrated in Fig. 4. In this architecture, the rule is extended as the in/out port, elastic label (i.e., channel spacing, grid, central frequency and spectrum bandwidth) and ODU label (e.g., tributary slots) which are the main characteristics of elastic optical networks. The action of optical node mainly includes four types: add, switch and drop a SDP to the port/label with specified adaption function (e.g., modulation format), and delete a SDP to restore the original state of equipment. Various combinations of rule and action are used to realize the control of elastic optical nodes and modulation format adjustment. The statistics function is responsible for monitoring the flow property to provide SDP provisioning.

 

Fig. 4 The extension of OpenFlow protocol for eSDN over elastic optical networks.

Download Full Size | PDF

3. Interworking procedure for eSDN over elastic optical networks

In this section, we illustrate the procedure for dynamic SDP setup/adjustment in eSDN over elastic optical networks.

Figure 5 shows the interworking procedure of eSDN for data center application in the proposed architecture. As shown in Fig. 5, two data centers are interconnected by elastic optical networks, while AC and TC control them respectively in eSDN architecture. For each SD-OTN node, once first access to the network, the handshake message of TCP session between TC and SD-OTN can be interworked to setup TCP connection preparing for the OpenFlow control. Each corresponding SD-OTN sends the OpenFlow hello message to TC actively at the initial phase, while obtaining the acknowledgment hello message replying from TC. After the control channel establishment, TC sends the node monitor request to each SD-OTN using OFPT_FEATURES_REQUEST message through OpenFlow protocol periodically. The corresponding SD-OTN responds the node and port information with OFPT_FEATURES_REPLY and OFPT_PORT_STATUS message respectively, so that TC can consolidate node information to discover the overall network node and topology information. If a new request of the data center application migration arrives, the TA-CSO strategy can be completed in AC and choose the optimal destination node after receiving the CSO request from TC, and then sends the result and application resource to TC. TC receives the CSO reply and computes a path with optimal modulation format in the network according to the transmission distance and optical network bandwidth information, and then proceeds to setup/adjustment an end-to-end SDP by controlling corresponding SD-OTN along the computed path by using OFPT_FLOW_MOD message. Differently with respect to packet-switched OpenFlow network, the elastic optical network can be needed to guarantee the end-to-end QoS. Therefore, all corresponding SD-OTN should report the setup status to TC through OFPT_PACKET_IN message. When TC obtains setup success reply from the last SD-OTN, the data center application can be migrated to the chosen destination node with the optimal modulation format for utilizing the application and spectral resources effectively. After that, the network information occupation in TC can be updated to keep synchronization by receiving the update message from corresponding SD-OTN via OFPT_PORT_STATUS message. The interworking messages described above are shown in detail on the right side of Fig. 5.

 

Fig. 5 Interworking procedure of eSDN for data center application.

Download Full Size | PDF

4. Experimental setup, results and discussion

To experimentally evaluate the overall feasibility and efficiency of the proposed architecture, we set up elastic optical networks based on eSDN for data center application comprising control and data planes based on our testbed, as shown in Fig. 6. In the data plane, four OpenFlow-enabled elastic optical nodes are equipped with Huawei Optix OSN 6800, each of which comprises flex ROADM and ODU boards and corresponding tributary card, making them possible to switch or transport the elastic signal in optical networks. We develop the software OFP agent according to the API function to control the hardware of flex ROADM and ODU boards through extended OFP. Data centers and the other nodes are realized on an array of virtual machines created by VMware software running at embedded Linux platform on IBM X3650 servers. Since each virtual machine has the operation system and its own independent IP address, CPU and memory resource, it can be considered as a real node. The virtual OS technology makes it easy to set up experiment topology based on the backbone of US which comprises 14 nodes and 21 links. Three data centers are deployed in various nodes of the experiment topology, which is shown in Fig. 6. In each data center, ten virtual machines are used to accommodate and provide the data center application. For OpenFlow-based eSDN control plane, the TC is assigned to support the proposed architecture and deployed in three servers for elastic spectrum control, physical layer parameter adjustment, PCE computation and resource abstraction, while the database servers are responsible for maintaining traffic engineering database, management information base, connection status and the configuration of the database and transport resources. The AC server is used for CSO agent and monitoring the application resources from data center networks with TA-CSO strategy. User plane is deployed in a server and applies the required application.

 

Fig. 6 Experimental testbed and demonstrator setup.

Download Full Size | PDF

We have designed and verified experimentally SDPs provisioning in elastic optical networks based on eSDN for data center application migration. The experimental results are shown in Figs. 7(a)-7(c). The destination data center is determined by AC with TA-CSO strategy based on various application utilizations among data centers and current network resource, and the SDP for the application migration is setup from source to destination node. Additionally, the spectrum bandwidth and corresponding modulation format can be tunable according to different SDP distances. The spectrum of various SDPs on the elastic link between two SD-OTNs is reflected on the filter profile, as shown in Figs. 7(a)-7(b). The end-to-end SDP setup/release/adjustment latency is measured through dozens of experiments as well as timing performance of the AC and TC with SD-OTN nodes, which comprises the strategy processing time of controller, OFP propagation latency and software and hardware of device handle times for the sake of observation and analysis, as shown in Fig. 7(c). The phenomenon can be seen in Fig. 7(c) that the strategy processing time of AC and OFP propagation time occupy an extreme limited proportion in the overall latency, which are around 2.5ms and 0.5ms respectively. We also observe that the relatively key contributor is signal action time in the device including device setup time, release time and adjustment time. The setup time (~55ms) is a little higher than adjustment latency (~50ms), since the Optix OSN 6800 will response a little slower with extra processing when a new SDP needs to be established. Moreover, the release latency is around 20ms and much lower than the latency of setup and adjustment, because the device does not need to reserve the spectral resources when the established SDP will be released. Note that, the time of device hardware handle is the main contributor of the overall latency, which is around 100ms on our testbed.

 

Fig. 7 (a, b) Bandwidth spectrum of SDPs. (c) SDP setup/release/adjustment latency.

Download Full Size | PDF

The experimental results are further emphasized in Figs. 8-9. Figure 8 verifies the status and destination node choice of data center application migration and various bandwidths of SDP in application interface of eSDN. Figures 9(a)-9(b) show the capture of the OpenFlow message exchange for eSDN through Wireshark deployed in TC. As shown in Fig. 9, 1.1.1.1, 1.1.1.2 and 1.1.1.3 represent the IP address of corresponding SD-OTN node respectively. When a SD-OTN connects to the network, the initial TCP session handshake will be performed through the interface to establish the TCP connection, and then SD-OTN sends the OFP_HELLO message to TC first. After receiving it, TC responds the OFP_HELLO message to corresponding SD-OTN and the control channel between TC and optical node is established. For the node information discovery, TC sends monitor request to each SD-OTN using OFPT_FEATURES_REQUEST message through extended OFP periodically, while obtaining the status information of each one with OFPT_FEATURES_REPLY message from them. After that, the overall network topology can be split joint in TC according to the port neighboring information through OFPT_PORT_STATUS message. Since the all optical nodes report the message simultaneously, Fig. 9(a) illustrates the capture of the OpenFlow message exchange of SD-OTN node with 1.1.1.3 for automatic network discovery procedure, which comprises control channel establishment, node and topology discovery procedure. After calculating the SDP with TA-CSO strategy, TC proceeds to set up an end-to-end SDP by controlling all corresponding SD-OTNs along the computed path by using OFPT_FLOW_MOD message. The experimental results correspond to the procedures we depicted in Fig. 5. Figure 9(b) shows a snapshot of the extended flow table modification message for SDP setup, which verifies the OPF extensions for eSDN over elastic optical networks and are the same as we described in Fig. 4.

 

Fig. 8 Application graphical user interface (GUI) of testbed.

Download Full Size | PDF

 

Fig. 9 (a)The capture of the OpenFlow messages for automatic network discovery. (b) The capture of extended flow table message for SDP setup.

Download Full Size | PDF

We also evaluate the performance of eSDN over elastic networks under heavy traffic load scenario, and compared TA-CSO with individual CSO and physical layer adjustment (PLA) strategies [9] through simulations. The traffic requests among data center nodes are established with spectrum randomly from 50GHz to 400GHz, where the adjustable minimal frequency slot is 12.5GHz. The service application usage in data center is selected randomly from 1% to 0.1% for each application demand. They arrive at the network following a Poisson process and results have been extracted through the generation of 1 × 10⁵ demands per execution. In the TA-CSO strategy, we set up the values of ${\text{R}}_a,{\text{R}}_b$ as 60% and 40% respectively according to the experience of experiments for simplicity. Figures 10(a)-10(b) compare the performances of three strategies in terms of blocking probability and resource occupation rate. As shown in the Fig. 10(a), TA-CSO strategy reduces blocking probability effectively than CSO and PLA strategies, especially when the network is heavily loaded. The reason is that TA-CSO strategy realizes global optimization considering both application and network resources integrally, furthermore on the basis of it, economizes the spectral resource again with choosing high-level modulation format according to the SDP distance. It also can be seen that PLA outperforms CSO strategy in blocking probability. That is because PLA strategy adjusts spectrum bandwidth through modulation format to save the spectral resource doubled, and increases greatly the available resources for new arriving demands. The phenomenon can be seen in Fig. 10(b) that TA-CSO strategy outperforms the other strategies in the resource occupation rate significantly. The main reason is that more resources can be occupied when the blocking probability is lower.

 

Fig. 10 (a) Blocking probability and (b) resource occupation rate among various strategies under heavy traffic load scenario.

Download Full Size | PDF

5. Conclusion

To meet the QoS requirement of data center application, this paper presents a novel elastic optical network based on eSDN architecture for data center application. Additionally, the TA-CSO strategy is introduced for eSDN in the proposed architecture. The main extended OpenFlow messages are described in this paper. The feasibility and efficiency of eSDN are verified on our testbed built by both control and data planes with four real OpenFlow-enabled elastic optical nodes for data center application migration. The setup/release/adjustment latency including strategy processing time of controller, OFP propagation latency and software and hardware of device handle time is estimated effectively. We also quantitatively evaluate its performance under heavy traffic load scenario in terms of blocking probability and resource occupation rate, and compare it with CSO and PLA strategies. The experimental results indicate that, in our tested scenario, eSDN with TA-CSO strategy can utilize cross optical network and application stratum resources effectively and dynamically adjust modulation format and bandwidth in elastic optical networks.

Our future works for eSDN include two aspects. One is to improve TA-CSO performance and extend the testbed to a large scale network topology with multi-domain. The other is to implement the network virtualization in data center interconnect with optical networks on our OpenFlow-based testbed.