An IEEE 1588 Performance Testing Dashboard for Power Industry requirements

The numerous time synchronization performance requirements in the Smart Grid necessitates a set of common metrics and test methods. The test methods help to verify the ability of the network system and its components to meet the power industry's accuracy, reliability and interoperability criteria for next-generation substations. In order to develop viable metrics and test methods, an IEEE 1588 Testbed for the power industry has been established. To ease the challenges of testing, monitoring and analysis of the results, a software-based testing dashboard was designed and implemented. The dashboard streamlines the performance testing process by converging multiple tests for accuracy, reliability and interoperability into a centralized interface. The dashboard enables real-time visualization and analysis of the results. The paper details the design and implementation of the IEEE 1588 Power Industry Performance Testing Dashboard as well as an update of the preliminary findings from the testbed.


INTRODUCTION
Enabling the next-generation automated substation's ability to gather multitudes of data from intelligent electronic devices (IEDs) will require sufficient contextual data quality. Improved data quality will minimize uncertainty when processing the data to establish situational awareness for more efficient and reliable substation control. The impact of data quality on distributed control algorithms over an asynchronous network have been shown to impact the quality of state estimation [1]. In order to merge the data from heterogeneous sources, accurate time-stamps play an essential role in detennining cause and effect. The network of substation end devices will require time synchronization with worst-case accuracy on the order of ±1 f.ls [2]. Reliable, high accuracy time synchronization continues to be difficult to achieve in complex systems [3]. Wide-Area Monitoring Systems (WAMS) can benefit from monitoring the accuracy of time synchronization and assessing the quality of the W AMS applications based on timing accuracy achieved [3]. Similarly, substation monitoring and control applications, which propagate infonnation to the wide area network, also need to have accurate time synchronization as one factor in achieving high quality control models by reducing measurement uncertainty. The  In order to streamline the testing process against the numerous requirements with respect to accuracy, reliability and interoperability, a software-based IEEE 1588 perfonnance testing dashboard has been developed. The dashboard, through a Graphical User Interface (GUI), enables perfonnance monitoring of the IEEE 1588 devices on the network, while providing centralized execution of the test methods, data visualization and analysis of the synchronization accuracy, reliability and interoperability. The dashboard is designed to readily integrate into any IEEE 1588-compatible network as it is based upon the Management Node messages in the 1588 version 2 standard [4]. The metrics used are based on industry requirements [1,5]. This paper introduces a novel means of enhancing the management node features to provide an automated testing dashboard for assessing conformance to the IEEE 1588 standard and IEEE 1588 power industry profile requirements. Additionally, the paper details test methods and results from new test scenarios including ring topology, ring topology link failure, traffic load, interoperability, and security. II.

PERFORMANCE CRITERIA
The dashboard test suites assess the perfonnance of IEEE 1588 devices on the network based where the performance criteria are accuracy, reliability and interoperability as depicted in Figure 1. The implementation currently focuses on the ability of IEEE 1588 devices to reliably maintain the required synchronization under a variety of plausible scenarios.
Factors impacting reliability include the implementation's capability to maintain synchronization over time in all conditions ranging from ideal, stressed to failure conditions in substation topologies such as linear, star and ring. Stressed OfficiaJ contribution of the National Institute of Standards and Tcchnology; not subjcct 10 copyright in the United Slates. Certain commercial equipment, instruments, or materials are identified in this paper in order to specify the experimental procedure adequate(v. Such illentijication is not intended to imp�v recommendation or endorsement by the National Institute of Standards and Technologv. nor is it intended to imp{v that the materials or eqUipment identified are necessari(v the best m>ailable for the purpose. conditions include traffic bursts on the network that could create packet delay variation (PDV), which significantly degrade the synchronization accuracy. Failure modes include loss of network connectivity, during which the substation must maintain synchronization of its network for as long as possible and as close to UTC (Coordinated Universal Time) as possible. The curr ent metrics used to assess the reliability of the synchronization include synchronization offset with respect to the GM, mean path delay between the GM and the OC, and out-of-specification probability of 10. 4 • Additionally, the number of security vulnerabilities is also considered a reliability metric.
Cybersecurity is pertinent to the Smart Grid. The National Institute of Standards and Technology (NIST) has developed guidelines for Smart Grid cybersecurity [6]. Therefore, reliability of the synchronization is also dependent upon the ability of the slave node and network to detect and to defend against cybersecurity attacks. Thus far, the dashboard includes test methods for Denial of Service (DoS), masquerade, delay, and multicast poisoning.
Interoperability among the IEEE 1588 nodes not only impacts accuracy and reliability, but also has implications on ease of system integration and interchangeability of substation devices. The IEEE 1588 standard specifies many requirements.
As a proof of concept, a few have been selected for the current testbed. Among the interoperability specifications that can impact performance, one IEEE 1588 parameter selected for testing is the synchronization (sync) interval. The evaluation method would include a scorecard of the required and optional functions for a specific IEEE 1588 node. Other interoperability metrics include ease of integration and ease of interchangeabil ity.

DASHBOARD DESIGN AND IMPLEMENTATION
The dashboard, shown in Figure 2, provides a centralized monitoring interface, automated means of executing test scenarios, visualizing the data in real-time, as well as real-time analysis statistics of the key metrics identified for the IEEE 1588 Power Systems profile [2]. The dashboard enables remote configuration of IEEE 1588 nodes in the network.

A. Management Node
The foundation of the test dashboard relies on the management messages. The IEEE 1588 management messages provide the ability to set and obtain data regarding the performance and status of the IEEE 1588 devices in the network. The management messages provide the ability to remotely and dynamically monitor and configure the network and each IEEE 1588 device.

B. Traffic Simulation
Another component of the testbed that is integrated into the dashboard is the traffic generator. The dashboard enables execution of the traffic generator through the GUI. In order to provide practical test methods, traffic loads representative of next-generation substations need to be simulated. As the traffic characteristics of next-generation substations are not yet available, the first set of simulations is based upon G.8261 Timing and Synchronization Aspects in Packet Networks [7].
The traffic patterns include static, square and ramp. The simulator can generate traffic at up to 100 percent of the network bandwidth using a specified traffic model. The objective is to inject traffic based on the IEC 61850 standard [8] and to simulate networks under heavy duress during a fault occurrence. It is expected that during a fault occurrence, the network will experience frequent traffic bursts. It is imperative to have good synchronization during a fault occurrence to be able to accurately correlate the cause and effect.

C. Graphical User Interface (GUJ)
The dashboard monitors the offset of synchronization and mean path delay between the Grandmaster and the ordinary clocks. To see the reliability over time, a histogram displaying the distribution of the synchronization offset is enhanced with color-coded outliers to determine the frequency of occurrence. When nodes are in peer-to-peer (P2P) mode, the delays between the peers are also displayed. The dashboard monitors the current status of the IEEE 1588 devices including the current elected Grandmaster and whether the ordinary clock is synchronized. The status of all the IEEE 1588 nodes, synchronization offsets over time, the distribution of the offsets, and mean path delays are visualized through the GUI in real-time as shown in Figure 2. The dashboard alerts the user when the offset approaches 75 ns and 100 ns, by color-coding the points yellow and red, respectively. The alert thresholds are configurable by the user via the GUI.

D. Test Execution
In order to ease the testing process to be able to automate the execution and repetition of the tests to provide data for analysis, the dashboard enables remote configuration of and automates the execution of the tests. The dashboard is capable of executing an entire test suite, a combination of different types of tests and parameter configurations. The following IEEE 1588 parameters and configuration variables can be set through the GUI: synclnterval, announce Interval, DelayReqinterval, DelayMechanism, and port status. Other test scenarios that can be executed via the dashboard include the security and conformance test methods, Grandmaster switchover, as well as network holdover and convergence. Data plots of the synchronization offset and mean path delay from the slave are also automatically generated for each test scenario.

E. Scalability through simulation
The simulation aims to incorporate virtual versions of common Smart Grid devices, specifically the PMU (phasor measurement unit), into the testbed. PMUs are becoming increasingly important in wide area monitoring and protection schemes. PMUs provide voltage and current phasor measurements to detect anomalies in the grid [9]. PMUs depend on synchronized time for accurate measurements.
Therefore accurate clock synchronization on the order of 1 !is of UTC is needed, which is within PTP capabilities. The simulation provides the ability to incorporate realistic synchrophasor traffic into the network [10]. IV.

PERFORMANCE RESULTS
The testbed is comprised of redundant PTP Grandmasters (denoted as GM1 and GM2) synchronized to the Global Positioning System (GPS). For the results described in this paper, we used four PTP switches with two different implementations. The PTP switches can be configured as Transparent Clocks (TCs) or Boundary Clocks (BCs). The PTP network currently has five ordinary clocks (OCs) configured in slave mode. OC2, OC3 and OC4 are based upon the same implementation. OC3 has an oven-controlled crystal oscillator (OCXO), while OC2 and OC4 have temperature-controlled crystal oscillators (TCXOs).

C. Holdover and convergence
The holdover tests provide a view of how the IEEE 1588 nodes would fare without a Master clock. The holdover durations tested include 10 s, 100 s, and 1000 s. With accurate time-stamping in the TC, the IEEE 1588 OCs were able to support holdover between 10 to 100 s while remaining within 1 f.l.s accuracy. Table 2 provides a sample of the synchronization offsets after the node establishes contact with the Grandmaster. OC3 holdover ranged from 200 ns to 2.5 f.l.S, whereas a less stable clock, OC4, drifted 448 ns in lOs to a drift of 4.7 f.l.S in 1000 s. OC5, which is compromised by a TC introducing a large timing error drifted significantly with a 2.6 f.l.S offset at lOs. At 1000 s, the maximum offsets of all three OCs went significantly above the 1 f.l.s threshold. It is important to note that since the dashboard relies on the offset responses from the IEEE 1588 slave nodes, it is currently not recording data when it is not synchronized to a Grandmaster. The dashboard will integrate the hardware measurement to be able to provide data during the holdover. In contrast to results from [5], the automation of test deployment enabled more data to be obtained on holdover and convergence patterns. Figure 9 indicates a consistent convergence pattern and duration within seconds over ten iterations, with an hour stabilization period. The holdover dispersion between runs indicates a large range of uncertainty in the behavior of the oc. While the pattern is consistent, the amount of drift can vary significantly. The variation is due to conditions such as ambient temperature, which can contribute to the variation in the drift rates. To address the issue of ambient temperature, using more robust quartz such as an OCXO would guarantee a smaller margin of error.
Analyzing and isolating the factors impacting the variation could ensure greater repeatability. However, the initial results indicate devices could benefit from robust shielding to be able to handle ambient conditions within the substations without adversely affecting the synchronization performance.  rome/minute) Figure 9. Convergence patterns from ten iterations for OC4 after the lOs holdover duration.

Security
The method of testing IEEE 1588 security is by exposing the network to attacks and detecting vulnerabilities. Several security vulnerabilities of the IEEE 1588 protocol have been identified [11], [12], [13] and [14]. In addition to verifying the existence of the required features, the dashboard provides additional analysis capabilities to verify the implementation performs to the specified configuration. For example, one interoperability test includes the ability to query the synchronization frequency available and then for each frequency determine the actual number of synchronization packets received within a specified window of time. Figure 10 displays the results from the synchronization rate where the left column is the specified log sync interval, the middle column show the rates and actual number of packets received when the test goes from a log interval of -3 to 3, and the last column show the rates and actual number of packets where the interval range is 3 to -3 to ensure the ability for rapid transition between interval specifications in both directions. Furthermore, the dashboard provides a prototype of how vulnerability testing can be developed and deployed. Though only a limited number of devices were available for test, by default, each node was vulnerable to at least some of the cybersecurity attacks. Therefore, it is imperative for the network administrator to ensure perimeter security for the IEEE 1588 devices in the network given the cybersecurity requirements of the Smart Grid [6]. To protect the network against these attacks, one solution is to implement Annex K of IEEE 1588 [6]. However, vulnerabilities have also been found and must be addressed [15]. A complete solution would be a secure protocol along with a security policy for the entire network [6].
Interoperability can also be a significant challenge to achieving the performance and reliability necessary to meet the power industry requirements. The dashboard implementation provides a prototype of how conformance testing can be executed via the Management Node messages in addition to profile requirements. In addition to verification of IEEE 1588 capabilities required in the profile, such as the accuracy requirement, the dashboard can also serve as a means to display the status of all the IEEE 1588-enabled based on the Management Base Information (MIB) Objects [2].
Future work on the test dashboard will include integration with the hardware synchronization offset measurement [5].
The focus will also include development of test methods for security, interoperability as well as adherence to the IEEE 1588 Power Profile industry requirements. Additional security tests, such as replay and delay attacks, as well as countermeasures will be implemented. The performance impact of the countermeasures will also be analyzed. A substation network simulation will also be integrated. The current IEEE 1588 simulation is limited in to replicating the effect of the synchronization protocol on each node's simulated local time. Future work will involve replicating the IEEE 1588 protocol down to each individual packet within the simulation. Along with the bridge between the physical testbed and simulation, this will allow the virtual nodes to act as IEEE 1588 slaves, exchanging synchronization messages with a real world grandmaster clock. The simulation would transition towards building a virtual substation network model synchronized with IEEE 1588. The testbed will also continue to expand to characterize new metrics impacting the performance criteria of IEEE 1588.