TCP Connection Management for Stateful Container Migration at the Network Edge

Container migration has emerged as the most effective way to ensure the proximity of time-critical microservices at the network edge with mobile end devices. However, ensuring service continuity while migrating microservices that rely on an established TCP connection is still a significant technical challenge. In this paper, we investigate such pivotal issue and propose COAT, a novel, yet simple, network architecture that leverages overlay network technology to achieve TCP connection migration. Through experimental validation using sample microservices, we show that, compared to the traditional container migration approach that does not support connection migration, our solution enables the successful migration of microservices relying on an established TCP connection, at the cost of a 14% maximum increase of the migration duration. Importantly, our solution to the problem of connection migration does not require the use of a dedicated protocol, or any modification to the application source code or the kernel.


I. INTRODUCTION
In recent years, edge computing has been acknowledged as the state-of-the-art paradigm to overcome the bandwidth and latency challenges in cloud computing architectures.The main idea of edge computing is to bring applications, computational capabilities, and storage facilities closer to the end users, thus significantly reducing processing and communication delays.Moreover, to fully exploit the benefits of cloud and edge computing architectures (namely, scalability, availability, and resiliency), applications are often designed in the form of microservices chains, taking advantage of the lightweight container virtualization technology [1].
Concurrently, due to the rapid development of mobile communication networks, the main consumers of edge services have evolved from static to mobile devices, such as connected cars and Unmanned Aerial Vehicles (UAVs)scenarios that require the support of high-demanding, latencyand bandwidth-critical applications.In this context, container migration techniques have gathered attention as an effective solution to address mobility challenges by ensuring continuous proximity of edge microservices with mobile devices.
Two fundamental migration strategies can be identified, i.e., stateless and stateful migration, with the second being used whenever keeping track of the microservice internal state is essential to guaranteeing service continuity.Importantly, despite the current trend favouring the development of stateless microservices, stateful microservices are still extremely common due to the complexity in refactoring legacy monolithic applications [2].
In this work, we therefore focus on stateful container migration and, specifically, on the problem of connection migration.Indeed, despite microservice migration is supposed to be ideally seamless, in practice, some service disruption must be accounted for, mainly due to (i) the traditional stateful container migration techniques, which require the freezing of the microservice state, and (ii) the migration of the microservice network connection with mobile end users.While in [3] we focused on the former issue, in this work we analyze the latter one.We underline that, despite several recent studies have experimentally demonstrated the potential and effectiveness of stateful container migration techniques, just few of them have investigated the akin connection migration issue.Moreover, such existing solutions are mostly application-specific and based on either kernel or protocol customization, thus it is impractical to integrate them with container virtualization technology.
To fulfill this gap and allow for a performant and efficient migration of stateful microservices, we propose COAT, a novel, yet simple, network architecture that, independently from the specific microservice, permits to preserve the established connection thereof with the mobile end users, during stateful migration.Specifically, due to its wide popularity and practical relevance, we focus on the TCP transport layer protocol [4], which by itself does not support the mobility of the connection endpoints.The benefits of COAT can thus be summarised as follows: • It migrates a generic microservice container with an established TCP connection; • It keeps track of the TCP connection states upon migration, thus avoiding reconnection procedures; • It preserves all the data queued inside the TCP socket, thus preventing data losses; • It performs the microservice stateful migration procedure in an agnostic way with respect to either the server or the client side of the connection.It is worth remarking that, to characterize and minimize the effects that contribute to service disruption, we also evaluate experimentally the migration performance when the COAT approach is implemented and validate our solution through real-world microservices.
The rest of the paper is organized as follows.Sec.II provides an overview of container and connection migration, thus introducing the main tools we used to design our solution.Sec.III describes the COAT network architecture and our enhanced version of the stateful container migration process.Sec.IV details the realistic microservices and the testbed setup we use in our validation experiments, which are presented in Sec.V. Finally, Sec.VI discusses some relevant related work while highlighting the novelty of our study, while Sec.VII draws our conclusions.

II. TECHNOLOGICAL BACKGROUND AND SOLUTION STRATEGY
This section presents an overview of container migration and the primary enabling tools to implement it.Further, it introduces the TCP connection migration challenges and the technologies we leverage to tackle these issues, namely, TCP_REPAIR mode and overlay networks.

A. Container migration
Container migration enables container relocation across hosts while meeting critical time constraints.Two fundamental container migration techniques have emerged: stateless and stateful migration.In this work, we focus on the latter, which is used whenever keeping track of the service state is fundamental to ensure service continuity.Hence, stateful container migration enables moving not only the container template image from source to destination host, but also the service internal state.In other words, the migrated container can seamlessly restore its previous working state, thus guaranteeing minimal impact on the Quality of Experience (QoE) of the final users.The fundamental off-the-shelf tools required to implement stateful container migration are CRIU and Podman, as detailed below.
CRIU [5].Checkpoint/Restore In Userspace (CRIU) is widely considered the key tool for stateful migration from a process layer perspective.It implements two major procedures: (i) the checkpoint procedure, which freezes a running process, collects its internal state, and encapsulates it into an image, and (ii) the restore procedure, which creates a new process and restores its state by leveraging a previously acquired checkpoint image.Such checkpoint image mainly includes: (i) the CPU-context state, e.g., the processes tree structure and the associated registers, (ii) the network sockets, (iii) the memory content, and (iv) the open file descriptors.Importantly, CRIU features the tcp-established option, which instructs CRIU to collect, along with the internal state of the container, the information related to the currently active TCP connection, thus allowing for a successful restoration of the TCP connection state during migration.
It is an open-source tool designed to develop, manage, and run containers and pods according to the Open Container Initiative (OCI) standards.Among the many offthe-shelf container engines, e.g., Docker and LXC, Podman is the one featuring the strongest integration with CRIU, by directly leveraging its APIs and, thus, effectively supporting container migration at the microservice layer.As a container engine, Podman enables the creation of isolated container environments by leveraging kernel namespaces and exposes the option to customize a container network namespace, thus providing high flexibility on the container network configuration.
Leveraging both CRIU and Podman, multiple stateful migration strategies can be defined.We focus on the most traditional and simplest one, i.e., Cold Migration, consisting of the following steps: (1) creation of a snapshot of the container (named "checkpoint") at the source host, (2) transfer of the checkpoint image from source to destination host, (3) restoration of the container at the destination host.

B. Connection migration
Connection migration is one of the critical issues concerning the migration of microservices with an always-established connection.We focus on TCP as connection-oriented transport protocol, as it is often used for legacy and modern edge applications [4], and we discuss the features of TCP that we can leverage to support connection migration.Notably, once a TCP connection is established, the protocol does not provide a way to modify or redirect such connection, unless through a complete re-connection procedure.To overcome this issue, and, hence, enhance the migration of TCP connections, a special option for the TCP socket has been introduced from Linux kernel version 3.5 onwards, namely, TCP_REPAIR [7].
When the TCP_REPAIR option is used, the TCP socket is switched into a special mode where any native TCP action performed on the socket has no effect (as depicted in Fig. 1).In this condition, the state of the TCP connection can be successfully "checkpointed" by CRIU and restored on a new host machine, with a probe packet being eventually sent to notify the other connection end point that the communication can be resumed.However, the TCP_REPAIR option is not widely used due to the following required conditions to achieve a successful connection restoration: (i) address consistency: the microservice container, when migrating from source to destination host, has to be assigned the same IP address; (ii) network reachability: when moved to the destination host, the microservice container must be able to directly reach the other end involved in the communication.In other words, the TCP_REPAIR option only provides the possibility to freeze and collect the state of the TCP socket, thus not tackling scenarios in which the IP address may change after migration.Moreover, to successfully resume the communication flow, the probe packet has to be correctly received at the destination, which is not trivial in the case of migration between distinct private networks.
We address the above requirements for TCP repair mode by defining a proper logical overlay network in which traffic flows can be dynamically managed.To do so, we leverage Open vSwitch (OvS) [9], a production-quality, multilayer virtual switch that provides two functions that are crucial for our purposes: (i) overlay network creation and (ii) network flow management.Indeed, it creates overlay networks based on Virtual Extensible LAN (VXLAN) -a technique that

III. COAT NETWORK ARCHITECTURE AND ENHANCED MICROSERVICE MIGRATION
We now present our solution, named Container OverlAy TCP (COAT) architecture, which effectively supports TCP connection migration and addresses the akin networking challenges by leveraging the previously introduced tools.Further, we integrate the COAT architecture in the container migration procedure, yielding an enhanced procedure referred to as COAT migration, thus enabling the migration of microservices that rely on an established TCP connection.

A. Reference Scenario
The COAT network architecture targets at supporting the simple yet crucial connection migration scenario depicted in Fig. 2. As an example, we consider a UAV as mobile end device, which connects to different stations (BSs) as it moves across the network.Due to the UAV's limited computational resources, some its critical functions (e.g., flight control with collision avoidance algorithm) must be deployed at the edge in the form of microservices and connected to the UAV using the TCP protocol.To minimize the experienced latency, such microservices should be deployed on the nearest edge server, i.e., the one co-located with the BS the UAV is connected to.We thus consider stateful container migration (see Sec. II-A) as the key technology to address such mobility challenge and ensure continuous proximity of edge microservices with mobile end devices.As thoroughly discussed in Sec.II-B, the problem of migrating the established TCP connection along with the microservice container is still to be properly addressed.As a solution to this issue, below we propose the COAT network architecture and the COAT migration process.

B. COAT Network Architecture
The proposed COAT architecture is depicted in Fig. 3, which includes three fundamental blocks, namely, the source host, the destination host, and the mobile end device.Source and destination hosts resemble the edge nodes that run a microservice before and after the migration process, respectively.The mobile end device, instead, is the node hosting the containerized client application that generates requests to be served by the microservice.The connectivity between the microservice and the client container is enabled by an overlay Fig. 3: COAT network architecture network implemented using interconnected virtual switches and customized network namespaces.
To effectively implement such architecture, we leverage the features provided by OvS to firstly create a virtual switch for each physical host and configure each of them to ensure their interconnection, thus defining the "backbone" of the overlay network.Secondly, we create two custom network namespaces, one for the microservice at the source host and one for the client container at the mobile end device.Both are then connected with the virtual switches, to complete the overlay network.Thirdly, we use Podman to run both the microservice and the client, and bind them to their dedicated network namespaces, hence connecting them to the overlay network.Once this third step is completed, the microservice and the client can communicate using the TCP protocol on top of the newly defined overlay network.
We underline that, when the microservice migration is performed, the TCP connection between the microservice and the client is preserved by (i) leveraging the TCP_REPAIR option to collect the connection state, and (ii) imposing an exact recreation of the microservice namespace at the destination host, especially in terms of its IP address configuration.Thus, COAT effectively solves the network address consistency problem since, thanks to the overlay network, the same IP address can be easily replicated at the destination host.Furthermore, since overlay networks enable the creation of a distributed network among multiple machines and to dynamically manage the traffic flows, direct reachability between microservice and client is always guaranteed, even after the migration process has been completed.Nevertheless, to effectively integrate the COAT architecture with the traditional migration process (see Sec. II-A), additional operations are required, which involve the creation and replication of customized network namespaces and the management of the flow control rules.

C. COAT Migration
To address such critical issues, we introduce COAT migration, which is an enhanced version of the stateful container migration process consisting of the steps illustrated in Figure 4 and described below.

Checkpoint container
Step 1

Clear network namespace
Step 2.1  4 may be executed in parallel, in order to speed up the migration procedure.However, as in this work we aim at validating the novel architecture and at assessing the impact of each step on the total duration of the migration process, in our performance evaluation all steps will be executed sequentially.
To summarize, COAT makes it possible to define an enhanced stateful container migration procedure to effectively support microservices that rely on an already established TCP connection.In particular, the proposed network architecture (i) allows for the migration of the TCP connection state, thus avoiding any reconnection procedure, (ii) preserves all the data queued inside the TCP socket, hence avoiding loss, and, (iii) does not require any modification at either the server or the client application to support a stateful migration.

IV. OUR TESTBED
In this section, we briefly describe the testbed we developed to validate the connection migration in COAT architecture and to assess the performance of the COAT migration procedure.

A. Microservices
We perform two independent sets of experiments using sockperf and iperf3 as examples of stateful microservices to migrate.They indeed resemble real-world microservices with established TCP connection and their features, briefly described below, allow us to effectively assess the network performance.Sockperf is a network benchmarking utility over socket API.It is a powerful tool to perform network latency measurements, which can provide a full log of each packet's transmission timestamps in sub-nanosecond resolution.In our testbed, we configure sockperf to measure the network latency using the TCP protocol, in order to assess the impact of COAT migration on the communication latency.Specifically, latency is measured at the sockperf client side through the so-called ping-pong test, which calculates the time difference between the timestamp in which a probe packet is sent to the sockperf server and the one in which the corresponding response from the sockperf server is received.
Iperf3 is a popular, lightweight tool for active measurements of the achievable bandwidth on IP networks.We use iperf3 to determine the impact of COAT migration on the communication throughput.To this end, we configure iperf3 to measure the throughput at the client side using the TCP protocol in reversed mode, i.e., the iperf3 server sending data to the iperf3 client, and we set the measurement interval to 100 ms.

B. Testbed and experimental setup
In our experiments, we leverage a cloud computing architecture featuring Intel Xeon Skylake CPU to instantiate three identical virtual machines (VMs).As shown in Figure 5, VM1 and VM2 represent two edge servers acting, respectively, as source and destination of the microservice migration process.Further, VM3 hosts the client container, thus acting as an end device that interacts with the edge servers.The migration procedure is controlled by a script running on VM3, which passes the control commands to VM1 and VM2 using the Secure Shell Protocol (SSH).Specifically, consistently with Figure 4, the commands corresponding to Step 1, Step 2.1 and Step 2.2 are executed at the source host (VM1), those related to Step 2.3 and Step 4 run at the destination host (VM2), while Step 3 runs locally at the end device (VM3).The three VMs can communicate with each other using their default network interfaces, i.e., eth1, eth2 and eth3 (resp.), which are provisioned by the underlying virtualization technology.
To emulate realistic values of network latency and throughput, specific queuing disciplines are applied to the default network interface of each VM using tc -a tool used to configure the Linux kernel traffic scheduler.Through these disciplines, the outgoing traffic of the network interface on each VM can be manipulated as needed.Consistently with the reference scenario presented in Sec.III-A, in our testbed we assume that VM2 has a closer physical distance to VM3 than VM1, thus resulting in lower and higher bandwidth experienced, which strongly motivates the need for microservice migration from VM1 to VM2.Therefore, in the latency experiment, we apply a 20 ms delay to eth1 and a 2 ms delay to eth2 and eth3.Similarly, in the throughput experiment, we limit the bandwidth of eth1 to 100 Mbps, and that of eth2 and eth3 to 500 Mbps.
The results shown in the following have been obtained by averaging over 50 runs, and computing the 90% confidence interval.

V. EXPERIMENTAL ANALYSIS
We now use our testbed under the settings introduced in Section IV to validate the COAT architecture and to evaluate the migration performance according to our enhanced stateful container migration procedure.
We start by looking at the TCP connection round trip time (RTT) (Figure 6) and the experienced throughput (Figure 7), before, during, and after a microservice migration.In particular, Figure 6 shows the RTT measured by sockperf as a function of the packet sequence number.To highlight the impact of the migration process on the experienced RTT, a measurement window of 160 samples has been extracted around such event, with packet with sequence number 80 being the one transferred during the migration.Consistently with the scenario we are tackling and the settings presented in Section IV, the experienced RTT decreases when the sockperf microservice is moved from the source to the destination host.Also, one can observe a peak in the RTT values corresponding to the time interval during which the migration procedure takes place.Interestingly, two considerations can be drawn: (i) despite the migration process, the packet transmission is successful (hence also enabling a correct RTT measurement), and (ii) the value of RTT measured for the packet transmitted in correspondence of the migration reflects the total duration of the COAT migration process.In summary, from the experiment we can conclude that (i) the TCP connection migration is successful, and (ii) no packet loss is experienced at transport layer during the migration process, which validates the proposed COAT architecture.
Figure 7 presents the temporal evolution of the throughput measured by the iperf3 client.In this scenario, the migration of the iperf3 server container is performed in the time window between 4 and 8 seconds.After the iperf3 server container is migrated, the measured throughput increases significantly, which is again consistent with our settings.Indeed, the end device can experience a larger link bandwidth with the destination host than with the source host (see Section IV).However, during the migration of the iperf3 server container, the iperf3 client experiences zero throughput, as evident from the plot in Figure 7.It follows that: (i) even though the client is unaware of the microservice migration process, it still experiences a service disruption, and (ii) the duration of such disruption notably corresponds to the total duration of the COAT migration process.In summary, in spite of the fact that the COAT migration can successfully migrate an active TCP connection, a service disruption during migration is unavoidable.Our next objective is therefore to thoroughly characterize the service disruption time, along with its components, and to identify which step of the migration process contributes the most to service disruption.
To this end, we perform an experimental analysis to break the migration duration measured by sockperf and iperf3 into different components.First, we notice that, in our experiments, the main migration control script runs on the end device, hence most of the commands to implement the COAT migration procedure need to be passed to specific hosts via an SSH tunnel.Using SSH tunnels to remotely control other hosts inevitably introduces an additional delay in the procedure.Thus, the total migration duration mainly consists of two contributions: (i) the SSH delay and (ii) the actual duration of the migration steps.
The SSH delay is presented in Figure 8, for each step Fig. 8: Additional SSH delay introduced to remotely execute the commands related to the COAT migration procedure of the migration procedure and for both the cases where sockperf and iperf3 are migrated.The duration of such delay varies from 500 ms to 800 ms, depending upon the specific network latency setting, the amount of transferred data (e.g., the command itself), and the exchanged certificates to ensure a secure connection with the remote host.Importantly, such SSH delay can be easily avoided by applying better remote control mechanisms -a relevant, interesting aspect that is, however, out of the scope of this work.
Next, the actual duration of each migration step is investigated in Figure 9.By comparing such duration for the two experiments with sockperf and iperf3 (resp.),one can observe that the values measured for sockperf are higher.This is due to the fact that sockperf is characterized by a larger microservice state size, mainly consisting of the memory allocation, which we measured to be, on average, 10 MB for sockperf and 1.5 MB for iperf3.Moreover, we remark that the duration of the checkpoint transfer step is affected by the network setting we described in Section IV.Interestingly, the fundamental migration steps, namely, checkpoint, transfer, and restore, dominate the COAT migration duration, and the duration of these steps is consistent with the results of our previous study on stateful container migration modeling [3].Thus, we can conclude that the COAT migration does not introduce any time overhead in the three most-impactful migration steps (checkpoint, transfer, and restore) with respect to the original process.Indeed, the COAT architecture only introduces three additional steps in the COAT migration process, namely, clear namespace, create namespace, and OvS flow update (see Figure 4), whose delay contribution is almost one order of magnitude smaller than that of the checkpoint, transfer, and restore steps.Remarkably, such additional steps are independent of the microservice state size; hence, their delay overhead would represent an even smaller percentage of the total overhead in case microservices with larger state size were considered.
In summary, we measured three duration metrics: (i) the COAT migration duration, comprising all COAT migration steps, (ii) the total migration duration, computed summing the above COAT migration duration and the SSH delay, (iii) the service disruption duration experienced by sockperf and iperf3 upon migration (also accounting for the SSH delay).
Notice that the SSH delay for the checkpoint step (i.e., Step 1 in Figure 4) is omitted from the calculation mentioned  in (ii), since it happens before the migration process starts.As reported in Table I, by comparing the total migration duration and the service disruption duration, for both the sockperf and iperf3 experiments, their difference is negligible, which means that the measurements provided by such experiments are consistent with our breakdown analysis.
In conclusion, compared to the traditional migration process (see Sec. II-A), we can safely conclude that the additional steps introduced by our COAT solution to enable connection migration determine an increase on the migration duration up to roughly 14% with respect to the COAT migration duration.
We argue that such overhead is reasonably small compared to the great advantage COAT provides in easily, yet effectively, supporting migration for microservices with established TCP connection.On the other hand, in the two scenarios we considered, the overhead due to SSH remote control represents approximately 50% of the measured migration duration.Hence, it is critical that, in place of SSH tunnels, more efficient remote control mechanisms are used to minimize the duration of a migration procedure.Importantly, we recall that the existing solutions are mostly applicationspecific and based on either kernel or protocol customization, which, as thoroughly discussed in the following related work (Sec.VI), are loose in generality, complex to implement, and impractical to be integrated with container virtualization technology.We advocate that these key points prevent an effective technical comparison with our solution.

VI. RELATED WORK
Stateful container migration has recently attracted a great deal of interest.An extensive survey on service migration in Multi-access Edge Computing (MEC) environments can be found in [10], while [11] presents an overview of current container migration techniques along with their fundamental metrics.However, far too little attention has been paid to the connection migration problem.Indeed, many studies, e.g., [12], [13], suggest to perform re-connection after a container is migrated.From a practical perspective, such an approach implies a customization of the client application source code to let it support the reconnection procedure.To the best of our knowledge, only few studies discuss solutions to enable connection mobility in a completely transparent manner for the client.Such solutions are mostly based on dedicated protocols, network proxy, overlay network tunneling, and software-defined networking (SDN).
The works in [14], [15] propose Multi-Path TCP (MPTCP) protocol as an effective solution to implement connection migration, since it permits to define multiple sub-flows for the same connection in a transparent way with respect to the client application.However, MPTCP requires kernel customization, implying practical limitations in real-world scenarios and unfeasibile integration with container virtualization technology.Similarly, [16], [17] thoroughly investigate the QUIC protocol and propose an extension thereof to effectively support server-side connection migration, thus advocating its validity for container migration.Despite the validity of this solution, it cannot be extended to other protocols, including TCP.Other approaches, e.g., the ones proposed in [18], [19], leverage the cloud platform's network proxy to hold and redirect active connections with external clients while performing intra-cloud or inter-cloud service migration.Likewise, [20], [21] design dedicated network proxies to redirect the network flows for general connection migration purposes.However, the use of centralized proxies is unfit for latency-critical edge computing scenarios since it breaks the proximity principle with mobile end users.Furthermore, [22] investigates the Locator/Identifier Separation Protocol (LISP), i.e., an overlay routing level on top of legacy IP, and suggests how to enhance it to effectively support VMs mobility management.However, this approach relies on a specific protocol customization, which limits the generality of the solution.In addition, [23] presents an SDN-based UAV controller migration testbed, which addresses the connection migration issue by manipulating the MAC addresses and leveraging SDN flow duplication functionality.Indeed, the design of this solution is tightened to a specific use case, i.e., UAV controller applications.
We finally recall that the main objective of our work is to enhance the stateful migration process to effectively support microservices with an always-established connection.We achieve this goal through an architectural solution based on overlay networks that, unlike previous solutions, is application-independent, requires no dedicated protocol and no modifications to the kernel or application source code.

VII. CONCLUSIONS
Container migration has become one of the fundamental technologies to support service mobility at the network edge.Nevertheless, multiple technical challenges still need to be addressed, especially those related to connection migration and service continuity.To fill this gap, we introduced COAT, a novel and effective, yet simple, network architecture that leverages overlay network technology to achieve TCP connection migration.To effectively integrate our proposed architecture with the stateful container migration process, we envisioned an enhanced migration procedure.We validated our solutions using popular, real-world microservices.In particular, our experimental analysis demonstrated that COAT migration process successfully enables microservice stateful migration while effectively preserving the state of the TCP connection, at the cost of a 14% maximum increase of the migration duration.We finally remark that COAT architecture effectively addresses the problem of connection migration without requiring a dedicated protocol or any modification to the application source code or the kernel.
Future work will focus on the integration of the COAT migration with orchestration systems, so as to minimize service disruption due to migration in more complex scenarios.

Fig. 6 :
Fig. 6: Sockperf migration experiment: RTT measurement as a function of the packet sequence number

Fig. 9 :
Fig. 9: Duration of each step of the COAT migration process, for the sockperf and iperf3 experiments Fig.4: COAT Migration, i.e., enhanced stateful container migration procedure integrating the COAT network architecture • Step 1: Checkpoint the running container at the source host using Podman with the tcp-established option.Both the microservice state and the established TCP connection state are now dumped into the checkpoint image and stop running.• Step 2.1: Clear the network namespace, thus preventing network configuration conflicts in the following steps.• Step 2.2: Transfer the checkpoint image from source to destination host.• Step 2.3: Re-create and configure the network namespace at the destination host to match the original one, which is required for the later container restore procedure to be successful.• Step 3: Update the network flow of the TCP connection.Firstly, update the flow control rule in OvS.During the network namespace recreation, a new virtual network interface is generated, along with a new MAC address.The ARP table at the client host is then cleared in order to ensure a successful ARP discovery process once the TCP connection is restored.
• Step 4: Restore the container from the checkpoint image.Now, the microservice, and its established TCP connection, can resume from its previous working state.We notice that Steps 2.1, 2.2, and 2.3 in Figure

TABLE I :
COAT migration performance using sockperf and iperf3