ECS-Grid: Data-Oriented Real-Time Simulation Platform for Cyber-Physical Power Systems

ECS-Grid is the first data-oriented real-time electromagnetic transient simulation platform for cyber-physical power systems (CPPS). Traditional simulation tools are constrained by object-oriented programming (OOP) architecture, which is now a significant obstruction to creating a comprehensive cyber-physical simulation. Therefore, the proposed ECS-Grid platform follows a new data-oriented paradigm based on an entity-component-system (ECS) framework, which delivers higher flexibility, extensibility, scalability, and performance to support cyber-physical system research. ECS-Grid proposes a layer of virtual intelligent electronic devices (vIEDs) to model IEDs in CPPSs. The vIEDs directly talk to physical components and communicate asynchronously with cyber services via the proposed high-performance JSON-like binary protocol. Tests with the islanding and the man-in-the-middle cyberattack scenarios on a 711-node ac–dc microgrid cluster based on a modified CIGRE 15-Bus system are performed and give accurate results. A faster-than-real-time performance is achieved on the 10th Gen Intel Core TM i7 computer, and real-time performance is achieved on distributed embedded NVIDIA Jetson platform. The ECS-Grid design and test results demonstrate the potential of the ECS data-oriented paradigm and may inspire the renovation of industrial simulation software.

ECS-Grid: Data-Oriented Real-Time Simulation Platform for Cyber-Physical Power Systems Tianshi Cheng , Graduate Student Member, IEEE, Tong Duan , Member, IEEE, and Venkata Dinavahi , Fellow, IEEE Abstract-ECS-Grid is the first data-oriented real-time electromagnetic transient simulation platform for cyberphysical power systems (CPPS).Traditional simulation tools are constrained by object-oriented programming (OOP) architecture, which is now a significant obstruction to creating a comprehensive cyber-physical simulation.Therefore, the proposed ECS-Grid platform follows a new data-oriented paradigm based on an entity-componentsystem (ECS) framework, which delivers higher flexibility, extensibility, scalability, and performance to support cyberphysical system research.ECS-Grid proposes a layer of virtual intelligent electronic devices (vIEDs) to model IEDs in CPPSs.The vIEDs directly talk to physical components and communicate asynchronously with cyber services via the proposed high-performance JSON-like binary protocol.Tests with the islanding and the man-in-the-middle cyberattack scenarios on a 711-node ac-dc microgrid cluster based on a modified CIGRE 15-Bus system are performed and give accurate results.A faster-than-real-time performance is achieved on the 10th Gen Intel Core TM i7 computer, and real-time performance is achieved on distributed embedded NVIDIA Jetson platform.The ECS-Grid design and test results demonstrate the potential of the ECS dataoriented paradigm and may inspire the renovation of industrial simulation software.

I. INTRODUCTION
T HE transition to clean and renewable energy in the power industry is playing a significant role in reducing the Tianshi Cheng and Venkata Dinavahi are with the Department of Electrical and Computer Engineering, University of Alberta, Edmonton, Alberta T6G 2V4, Canada (e-mail: tcheng1@ualberta.ca;dinavahi@ualberta.ca).
Tong Duan is with the National Digital Switching System Engineering and Technological Research Center, Zhengzhou 450001, China (e-mail: tduan@ualberta.ca).
Color versions of one or more figures in this article are available at https://doi.org/10.1109/TII.2023.3244329.
Digital Object Identifier 10.1109/TII.2023.3244329emission of greenhouse gases and fighting climate change [1].However, the traditional power systems that rely on centralized control networks and controllable generators become insufficient to meet the challenges of future power systems such as microgrids with the high penetration of uncertain and unstable renewable energy [2].Therefore, new intelligent decentralized control solutions based on modern information and communication technologies are emerging to face the new challenges [3], [4], [5].The new research works heavily involve communication between control centers and intelligent electronic devices (IEDs), which are the foundation of smart grid and power system automation [6].Thus, detailed and accurate real-time simulation of cyber-physical power systems (CPPS) [7] is necessary for future power system research [8], [9], [10].However, the scalability, flexibility, and performance of the traditional power system analysis tools and communication analysis tools are inadequate.For example, the existing software-based simulation approaches such as EPOCHS [11], GECO [12], INSPIRE [13], and the simulators proposed in [14], [15], and [16] aim to create a network interface for existing power system simulation tools and glue the two domains in power grid simulator and communication network simulators (NS-2, NS-3 and OPNET et al.), which cannot reflect the behaviors of IEDs in real-time environments and must handle the complicated synchronization between two different simulation domains.Some works such as [17], [18], [18], and [19] bridged the commercial real-time power system electromagnetic transient (EMT) digital simulators to the communication simulation systems, which can achieve real-time performance and more realistic behaviors.However, the highcost commercial EMT simulators were designed for industrial verification purposes and still lack the scalability and flexibility for CPPS-related academic research.
The essential problem is that both power grid and communication network simulation tools are initially designed for their single domain.The traditional software mainly based on objectoriented programming (OOP) has become a huge obstruction to building a native and comprehensive cyber-physical simulation platform.The industrial programs are dealing with various forms of data and their combinations, while the OOP paradigm emphasizes predetermined inner structure and relationships of objects.Plain data such as an array of float numbers can represent many things in the computer world.On the opposite, a class and its object can only be used for one purpose predefined by abstract templates, which brings significant difficulties to repurposing existing designs, and thus, cannot elegantly describe and solve the problems in the complex interdisciplinary CPPS.
Therefore, this article proposes for the first time, the ECS-Grid: a novel real-time cyber-physical EMT simulation platform with virtual IEDs (vIEDs) based on the cutting-edge entitycomponent-system (ECS) software framework.The proposed ECS-Grid simulation platform has the following major advantages.
1) High flexibility: Compared to the traditional dominating object-oriented paradigm which is based on polymorphism, abstraction, inheritance, and encapsulation, the ECS framework is based on a data-oriented paradigm: entity (usually an integer), component (pure data structure), and system (plain functions to perform algorithms on components), where entities are defined by the combination of data components, and component functionalities are defined by systems.This data-oriented paradigm avoids dependence complexities caused by OOP inheritance and brings flexible model description ability.Any data component and the system can be replaced not only at the compiled time but also at the run time.Such a feature is highly desired for cyber-physical simulation since the various forms of data and data flows are the major concerns.For example, real-world IEDs are composed by multifunctional circuit boards and these replacable boards can be represented by data components on a vIED entity.2) High extensibility: With the advanced data-oriented design, components and systems are grouped into plugins in ECS-Grid, and a simulation application is composed of a set of plugins.In contrast to traditional software which often provides a huge library as an undividable whole, ECS-Grid allows users to only pay for what they need.
Although it is initially designed for CPPS simulation, it can run pure physical simulation similar to simulators without cyber layers, or run the cyber features for other purposes without the physical EMT simulation.Moreover, the users can create plugins easily even in dynamic libraries with their customized components and systems, and add or override core functionalities such as the matrix solvers or additional communication protocols.3) High scalability: With the benefits from the ECS framework, a vIED layer is proposed which mainly utilizes scalability protocols from the message-oriented asynchronous ZeroMQ [20].A MessagePack-based [21] JSON-like simulator protocol is proposed for the simulator to bridge various industrial protocols.The utilization of the middleware makes it easy to scale ECS-Grid from a single CPU node to multithread applications or even distributed networks which resemble real-world automation systems.The performance test on a single-thread ZeroMQ vIED with MessagePack-based protocol shows a minimal latency of 6 μs and an average latency of 20 μs with an effective 60 Mbit/s bandwidth, which is quite enough for a wide range of application scenarios.A 711-node ac/dc microgrid cluster based on the modified CIGRE-15 Bus microgrid system with a man-in-the-middle cyberattack scenario is set up for demonstration and performance evaluation.The simulation results show that the proposed solution can achieve faster-than-real-time (FTRT) performance on 10th Gen Intel Core i7 CPUs and real-time performance on NVIDIA Jetsons with dual-core ARM v8 CPUs.
The rest of this article is organized as follows.Section II introduces the fundamental architecture and methodologies in ECS-Grid.Section III introduces the simulator MessagePack protocol and the performance test of vIEDs with different transportation; the way to implement industrial protocols such as IEC-60870-5-104 is also discussed.Section IV presented the microgrid cluster study case with results from one steady-state scenario and one cyberattack scenario.Finally, Section V concludes this article.

A. Data-Oriented ECS Architecture
The information exchange between physical and cyber systems is the major concern in a cyber-physical simulation.Information is carried by data, and generally, cyber systems are built to transport and process data that carry useful information.However, while data can represent almost anything in the digital world, an object which contains both data structure and behaviors can only carry limited information whose pattern is predefined by its abstract templates: the Class, without the ability to mutate its structure.The Inheritance makes it even worse due to the extra dependencies between Classes.As shown in Fig. 1(a), the OOP paradigm creates abstract base Classes for different domains while inherited implementations are realized in sub Classes.This adds difficulties in refactoring and optimization.Since cyber-physical systems in the Big Data age are transporting enormous unstructured data, a data-oriented solution that focuses on data processing and data combinations is highly preferred to OOP solutions.
Data-oriented programming means data combinations determine functionalities, which is also the core concept of ECS-Grid.The ECS framework starts to play a significant role in the game industry and modern software engineering, which is now the backbone of Minecraft [22], Data-Oriented Technology Stack (DOTS) in Unity [23], Call of Duty: Vanguard, ArcGIS Runtime SDKs by Esri, and many modern large-scale commercial software projects.However, it is still not utilized for cyber-physical simulation in power industries which are full of data-intensive applications.Currently, there are three major types of ECS frameworks: bitset, archetype, and sparse-set, where sparse-set is the most popular one due to its high flexibility and archetype has the best theoretical performance.The specific types of components are managed by an entity registry to provide database-like access to the data objects.In this article, the sparse-set-based EnTT [22] is used as the entity registry, which is also used in Minecraft.Everything under the ECS framework belongs to an Entity, Component, or System.The inheritance is replaced by an entity's composition of data components in the ECS framework.An Entity is an integer identifier that is linked to multiple components in an entity data registry which can be seen as a data table in Fig. 1

(b).
A Component is a structure with data to process.As shown in Fig. 1(b), entities are rows of the data table.For example, a voltage-source converter (VSC) entity is composed of the four circles in a row of the table view; the EMT model object which is the same circuit object of the traditional OOP design such as an averaged-value model VSC; the IO module which holds measured signals and controller signals; the VSC controller holds data for control logics and the IED communication module holds the information of network sockets and other parameters such as latencies.
A specific combination of data components will be processed by relevant Systems which contain all program logic.A System is a plain function that can process columns of components in the table of Fig. 1(b).It usually takes the registry as the input parameter and creates a query view of components from the registry just like a database query.The queries are optimized by the ECS framework depending on its storage type and usually are blazing fast.Because the system function is only called once on specific types of components, it eliminates the bloating issue caused by intermediate interfaces or CPU overhead in dynamic virtual function calls per object; since the components are stored in compact arrays, it is also more cache-friendly and easier to take advantages of modern single-instruction-multiple-data (SIMD) hardware such as graphical processing units (GPUs).
In other words, it can fully avoid the usage of inheritance and polymorphism to build more complex and efficient software with an ECS framework.Also, ECS brings impressive flexibility to modify an Entity.For example, one can replace the VSC control component without breaking other VSCs with the same physical model, or replace a system at the runtime to change the functionality.
The proposed ECS-Grid currently uses a hybrid ECS solution to fully reuse the traditional OOP EMT simulation code.The EMT simulation loop is untouched, and no modification is added to any physical component class.The only difference is the traditional physical components are now managed as a part of an entity in an ECS registry instead of an all-in-one object.The IED feature is added by introducing new components and systems to the physical software.This hybrid solution can be very useful for industrial developers to transfer from traditional OOP to data-oriented design under the ECS framework.The full transition to an ECS data-oriented simulation framework requires many critical changes to traditional design patterns and still needs some exploration.Details about EMTModel are discussed in Section II-B.IED components and systems are discussed in Section II-C.

B. Physical EMT Simulation
Currently, only EMT physical simulation is implemented in ECS-Grid.In EMT simulation, dynamic physical components such as capacitors and inductors are represented by differential equations.Small and simple circuits such as RC, LC, and RLC can be solved with a form of ordinary differential equations or state-space format.However, a power grid usually consists of a large amount of different physical components, which is more suitable to solve as differential-algebraic equations (DAEs) with nodal analysis.The nodal analysis is based on solving a circuit equation system with nodal voltages as primary unknown variables.Previous research works often emphasize elementwise derivation, while in this article the simulation is purely expressed by the language of linear algebra.Using Kirchhoff's current law (KCL), any RLC circuits with k nodes (ground node excluded) can be represented by where B is the oriented incidence matrix whose rows are corresponding to the physical components and columns are corresponding to nodes, L is the inductance, C is the capacitance and G is the admittance, s is the vector of current injections by sources.B is a transformation to gather the port voltages from global nodal voltages v, while B T can scatter the branch currents into the nodal injection vector.The W matrices are the weighted Laplacian matrices of different types of components, which are also called admittance matrices and play important roles in solving power grid equation systems.
[X] means diagonalized matrix of 1-D vector X.
To solve (1) with the Trapezoidal rule, the following equations can be obtained: Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.
where I n+1 Leq and I n+1 Ceq are synthetic current sources created by discretization.A similar rule is applied to all other dynamic or time-varying physical components in the power system for EMT simulation, which gives the general form where Y indicates the final admittance matrix for the system solution, which can be inversed to solve the primary unknown variable v n+1 ; I n+1 eq indicates all equivalent current sources generated by physical components; W dif f is the admittance matrix derived from differential equations such as Δt 2 W L for inductors.
For nonlinear components such as diodes that have nonlinear voltage-current characteristics.Equation ( 4) is extended to where f is an elementwise nonlinear function of B N v n+1 .Newton's method is utilized to linearize the system and solve it, which converts (5) into the following: where m denotes the iteration index of the Newton method, J m is the system Jacobian matrix at mth iteration.Equation ( 6) can be reorganized into where Therefore, the nonlinear components have the harmonized format of admittance matrices and artificial current injections, which gives the following recursion formula: The Jacobian matrix needs to be assembled and inverted serval times in each simulation time-step.Therefore, the f of many nonlinear components may be converted into piece-wise linear function or use v n to approximate v n+1 to speed up the computation.Equations ( 1)-( 9) cover the fundamentals of the physical power system EMTModel components and systems in ECS-Grid.

C. Data-Oriented IED Simulation
In most CPPS simulations, there were only two layers: the physical layer and the cyber layer, which often ignored firstclass citizens in real-world CPPSs: intelligent electronic devices (IEDs).The IEDs are generally protection, control, and monitoring devices in power grids, which are cornerstones for the modern power system automation and smart grid [24].Therefore a comprehensive cyber-physical simulation should model these IEDs to be as realistic as possible.
As shown in Fig. 2, a commonly used IED in power systems is composed of multiple modules: DSP controller, CPU, Network DSP (only for optical IEC-61850 GOOSE/SV), AI, AO, DI, DO for analog or digital inputs/outputs (IOs), and the power source, which are circuit boards is responsible for specific tasks.Different configurations of the board modules and internal firmware will define the functionalities of the IED.It can be beneficial to model the IEDs inside EMT simulation to reflect real-world communication behaviors such as network latencies, time synchronizations, and protocol analysis, and also bring many new possibilities to cyber-physical research.The structure of real-world IEDs is a perfect match for the proposed data-oriented architecture.
ECS-Grid proposed the layer of vIEDs, which is an independent set of components and systems to model IEDs in power grids to conduct control or communication tasks.These digital twins of real-world IEDs bring more realistic and more consistent experiences from real-world CPPS.The vIEDs consist of IO, control, and communication components in the ECS-Grid.This covers the fundamentals of a real-world IED in Fig. 2.
As shown in Fig. 3, the components of VSC entities in Fig. 3 are similar to the modular boards in Fig. 2; The systems:VSC_IO, VSC_Control and VSC_IED are similar to the Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.software programmed into the physical IED; the systems are grouped and called in the IED Stage which is considered as a new sensing layer compared to old physical simulation loop.Similar to the replaceable boards and upgradable programs in real-world IEDs, the components and systems of vIEDs can be replaced or reorganized to serve different purposes both at compile-time and runtime.This is realized elegantly within a data-oriented ECS framework while OOP cannot compose it nicely due to its fixed predefined structures.The vIED extension is implemented by a very simple plugin interface introduced in Section II-D.

D. Plugins Made Easy
The extensibility is important for a cyber-physical simulation platform and that should be a significant advantage of a dataoriented design.The functionalities of ECS-Grid are defined by a combination of plugins, which is similar to many popular ECS frameworks such as Flecs and Bevy.Plugins can have various inner structures and definitions as long as they provide a plain function with a declaration of void build(World &world); as the entry point.In this way, a plugin with functionalities in Fig. 3 can be loaded from a header-only library, a static library, and even a dynamic library loaded at run-time.The implementations are quite straightforward and an example C++ header of the ZeroMQ vIED plugin is included in Appendix A.
As shown in Fig. 4, a simulation based on the ECS framework is composed of various plugins, which is flexible and bloat-free.For example, although the solver and vIED plugins are the same, the simulation for microgrids uses exclusive microgrid systems such as renewable energy sources and storage units along with the droop controllers, while the high-voltage direct current (HVDC) simulation configuration only uses the MMC and Bergeron line model plugins.To test the vIED only, the physical plugins are removed and replaced by dummy data sources.These configurations are practically applied to produce the results presented in Sections III and IV.

III. PROPOSED DATA-ORIENTED PROTOCOL FOR REAL-TIME CYBER-PHYSICAL SIMULATION
Traditional CPPS simulations are usually based on available commercial simulators, where the signals of the physical power grid are grouped, converted to industrial protocols, and sent to cyber simulation machines.However, the industrial protocols are designed for production environments which should consider security issues, standard requirements, guidelines of power system operations, and the limits of existing industrial communication routes and devices.However, the simulation environments should provide a more generic protocol to simulate various scenarios which cannot be covered by a single industrial protocol.Also, the simulator itself should provide exclusive remote control and management functionalities for simulation-only purposes which are not considered by industrial protocols and IEDs.
Although some platforms [14] use Open Platform Communication (OPC) or CORBA (Common Object Request Broker Architecture) DIM (Distributed Information Management) protocol to unify the protocols within the simulator, these traditional OOP protocols are based on the late 1990s standards and technologies which cannot meet the data-oriented demands of modern cyber-physical simulation.The OOP protocols often need a cumbersome object library to decode the messages and many functionalities are fixed.Therefore, a data-oriented protocol and a local simulation network are proposed for the vIEDs as a unified middleware interface to the outer systems.The data-oriented protocol should be as follows.
1) Generic: It should be able to represent different messaging patterns used in microgrids and not be restricted to specific transport media.2) High-performance: It should not add heavy overhead to the simulation systems and can handle a large volume of data; it should have distributed and concurrent features to make full use of modern hardware.3) Customizable: Unlike industrial protocols where all are defined by standards, the CPPS simulation should enable more possibilities for research explorations of future power systems by allowing users to customize the protocol.

A. MessagePack Format for User Applications
The ZeroMQ mainly abstracts the sockets for higher level applications, the payloads being transported depends on the user's decision.The default vIED plugin uses a simple solution based on javaScript object notation (JSON) and MessagePack is proposed for a generic and customizable application-layer Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.simulation protocol.JSON is the first-class data format inside ECS-Grid for configurations and data exchanging shown in Fig. 5.It is faster and smaller than the current XML format used in industrial applications [25].The JSON format is selfdescribing, so there are no complex data models predefined by an Interface Definition Language (IDL) decode the messages.MessagePack is an efficient binary serialization format and it can exchange JSON data cross multiple languages more efficiently [21].The receivers can easily decode the messages to JSON objects like dictionaries in Python, ECMAScript, and Rust and handle them in their program logic.The utilization of MessagePack can provide a faster serialization and deserialization speed without a predefined schema and reduce the size by more than 40% compared to a plain-text JSON message.The example for the JSON-like protocol and the protocol conversion.The MessagePack design enables users to simulate specific scenarios and make custom virtual cyber services such as microgrid control center (MGCC) upon vIEDs, which can provide a very convenient platform for developing future distributed multilayer control schemes and other cyber components as shown in Fig. 6.
Real-time communication performance is easy to achieve since simulation environments have much better computing power, bandwidth, and reliability than field devices.Modern CPUs have quite a large memory bandwidth that is larger than high-end optical networks.For example, Intel Core X-Series Processors can achieve a bandwidth of 94 GB/s with 4-channel DDR4 2933 Mhz memories [26], which is nearly eight times faster than high-end 100 Gbit/s Ethernet.The current 10/100 Mbps industrial Ethernet bandwidth is no match to the CPU's internal bandwidth.Although the protocols are useful for simulation environments, the industrial protocols cannot be ignored.As shown in Fig. 5, protocol converters are the solutions, which map the MessagePack protocol to a specific protocol such as IEC 60870-5-104.Protocol converters are common in real-world power automation systems and many IEDs can do protocol conversions internally according to the firmware or hardware configurations.Thanks to the multitransportation ZeroMQ, the protocol conversion can have multiple choices to meet users' demands and interoperability can be ensured by customizing MessagePack messages.If users want industrial protocols directly built into the IED, they can follow the same plugin development principles to integrate their protocols.

B. Comparison of Various Middleware Protocols
Currently, there are three communication plugins available for vIEDs in ECS-Grid: ZeroMQ [20], eProsima Fast DDS Real-Time Publish-Subscribe protocol (RTPS) [27], and Eclipse Paho MQTT [28].Fast DDS is the middleware used in Robot Operation System 2 (ROS2).The MQTT is used for Internetof-Thing (IoT) applications and partly in microgrid applications with IoT devices.ZeroMQ is a widely used message-oriented middleware.The latency test results of different protocols under the one-publisher-one-subscriber vIED scenario are listed in Tables I-V.
Table I shows the results from the Fast DDS RTPS protocol.There is a spike in maximum latency when the message number increases, which is normally due to unreliable user datagram protocol (UDP) transportation.In summary, RTPS's performance is high and stable, and it has advanced features   which can be very useful for vIED applications.However, it requires many dependencies, and the provided advanced features are not used in power systems.Moreover, it is not easy to use and the support documents should be greatly improved compared to other solutions.
Table II shows the results from Eclipse MQTT Paho clients.The MQTT is not designed for microsecond-level latency, and it must have a broker, which is an Eclipse Mosquitto broker [29].The default configuration also enables message persistence on the broker server.Therefore, the latency is 1-100 ms level which is good for most IoT applications but not good for low-latency communications.However, the bandwidth reaches the top of all protocols when the published message number is 1000.In all, the MQTT solution can be useful for some IoT scenarios since not all devices need microsecond-level latency.
Tables III-V show the ZeroMQ vIED performance under different configurations.The in-memory inter-thread communication reaches the lowest latency of 6 μs, which is quite enough for IEC-104 applications since the protocol time-stamp has a resolution of milliseconds.The ZeroMQ point-to-point TCP pub/sub latency is around 20 μs and the TCP pub/sub with a broker test is just a doubled point-to-point TCP latency.The single-thread publisher's bandwidth is high and stable without throughput optimization and well suited for a real-world IED which mainly has a 100 Mbit/s Ethernet port.Besides, ZeroMQ is quite flexible and easy to use in every major programming language.The only problem is it requires more user decisions to establish an in-production network; however, it is an advantage for a simulator that can give users the maximum freedom to establish customized scenarios.
For the generic and high-performance goals, ZeroMQ is recommended to be the message bus between vIEDs.ZeroMQ is a high-performance asynchronous messaging library, aimed at use in distributed or concurrent applications.ZeroMQ supports scalability protocols (pub/sub, request/reply, client/server, and others) over a variety of transports (TCP, in-process, interprocess, multicast, WebSocket, and more).This keeps the code clear, modular, and scalabe from very low-latency in-memory communication to the large-scale cloud computing scenario.

IV. CASE STUDY, RESULTS, AND PERFORMANCE
Fig. 7 shows the microgrid cluster connected by a multiterminal dc system, which forms a 711-node power system with 60 vIEDs to evaluate the proposed simulation platform's functionalities and performance.The microgrid is a 15-Bus power distribution system derived from the CIGRE report [30] and pandapower [31] case files; loads are reduced to 10% and a 5MW Li-ion battery storage is added to Bus-1 to ensure the ability of islanded operation.The microgrid has 16 VSC stations Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.and they are all modeled by the average-value model to reduce the complexities of control and simulation.Each VSC station has a VSC controller and a vIED acting as a remote terminal unit (RTU) to the VSC station.Each battery storage has an extra vIED to control the battery charging.The distributed power sources are controlled as PQ nodes which have fixed power generations, while the storage stations are controlled by a droop controller to auto-balance the system and provide a stable frequency.The loads are modeled by fixed RLC components for convenience.Three modified CIGRE 15-Bus microgrids are connected to the three-terminal HVDC system.The ±50 kV HVDC system consists of three 51-level 50 MW modular multilevel converters (MMCs) and MMC-1 is designated to control the dc voltages.The other two MMCs are set to drain 1MW from the HVdc system.The MMCs are modeled by detailed-switching models which means voltage balancing of submodules is needed.In this article, the nearest-level modulation is used for MMC's lower-level controller.The upper-level controllers for MMCs are similar to VSCs in microgrids which control the dc voltages or the power generations.
Fig. 8 shows two scenarios for the test results.Droop_0 IED in MG-1 is the main research target.Scenario 1 is used to evaluate the islanded microgrid clusters and produce the steady state for Scenario 2. Scenario 2 conducts a man-in-the-middle cyberattack to manipulate secondary frequency regulation command and cause catastrophe across the cluster.Scenarios 2 is similar to the real-world industroyer cyberattack in 2016 and industroyer2 attack in 2022 on Ukraine power grids [32], which hijacked supervisory control and data acquisition (SCADA) systems and sent dangerous commands to IEC-104 RTUs and IEDs.
Fig. 9 shows the setup of the real-time hardware platform introduced in Fig. 6.The three NVIDIA Jetson AGX Xavier embedded computers with real-time Linux installed are used to simulate physical microgrids, the corresponding MMC station, and vIEDs.The Xilinx VCU118 board is used to handle fast signal IO to support hardware-in-the-loop functions.The PC server runs cyber services such as virtual MGCC, IEC 60870-5-104 clients, and cyber simulation tools.

A. Results and Performance
The results of Scenario 1 are shown in Fig. 10(a)-(f).Fig. 10(a)-(c) is the frequency, real power, and bus voltage waveforms of VSCDroop_0 storage station in each microgrid, respectively; when t < 15 s, the frequency in MG-1 F Droop1 is 50 Hz since the ideal three-phase ac source is attached to MG-1 Bus-0, while the other microgrids have different frequencies; at t = 15 s the ideal source is removed so that a large deviation occurred to F Droop1 , while MG-2 and MG-3 have no obvious changes because the dc system can allow asynchronous frequencies; the steady-state value of F Droop1 is 50.02 and P Droop1 is 4.0 MW (0.8 p.u.), which meets the droop control equation F − F ref = K p (P ref − P ) = 0.02.This is considered a primary frequency regulation process.Fig. 10(d)-(f) is the dc voltage, real power, and reactive waveforms of MMC stations, respectively; the rated dc voltage is 100 kV so that all MMC stations maintained a nominal voltage according to Fig. 10(d  in Fig. 10(h).However, these drastic deviations are measured PLLs and may not reflect the real situation in the physical systems under faulty conditions and that is why real-world power systems also have digital fault recorders to record the EMT waveforms when faults occurred.Fig. 10(i) shows the EMT voltage waveforms of Bus-1 captured by virtual digital fault recorders, which are triggered by the fault detection mechanism in the proposed simulation platform.The EMT waveforms revealed the physical details that happened after t = 147 s; the system started to react about 1 s later than receiving the hacked message, and the drastic deviations in measurements may be caused by the high-frequency oscillation which can affect stability.The simulation of Scenario 2 shows the catastrophic consequence of cyberattacks in a vulnerable power cyber network.
Besides the real-time Jetson platform, Table VI shows the performance of the proposed cyber-physical simulation platform on an x86 machine (Intel Core i7 10700 k 8c16t@4.7 GHz, 32 GB DDR4 3000 MHz, Ubuntu 20.04, GCC 11.1).The pure physical parallel simulation consumes 7.87 s which is 5.08 times faster than real-time, and all cyber-physical simulations also achieved faster-than-real-time (FTRT) performance even with 100μs ticking interval.For the millisecond-level communication intervals, the cyber-physical cosimulation can achieve high efficiency and the overhead is almost deflectable.The overhead can be further reduced with more concurrency for socket polling, data encoding, and decoding since the communication systems currently execute in series inside the main simulation loop.The FTRT functionality can enable predictive and preventative control actions in energy control centers.

V. CONCLUSION
ECS-Grid was a novel data-oriented cyber-physical simulation platform for microgrids under the ECS framework proposed to model the IEDs in a power system with flexible data components and extensible plugin architecture.Furthermore, a modern JSON-like MessagePack-based protocol was proposed for the vIEDs and was capable of completing various tasks needed for cyber-physical transient simulation.The results from the scenarios in the microgrid cluster study case showed the accurate system behaviors and real-time or FTRT performance of Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.
ECS-Grid.The IED systems and components could be extended to cyber-physical power dynamic or steady-state simulations, thanks to the data-oriented design.The data-oriented ECS-Grid could inspire the renovation of industrial software tools and boost further research of the future CPPS.

Manuscript received 1
August 2022; revised 19 November 2022; accepted 31 January 2023.Date of publication 13 February 2023; date of current version 19 September 2023.This work was supported by the Natural Science and Engineering Research Council of Canada (NSERC).Paper no. TII-22-3342.(Corresponding author: Tianshi Cheng.)

Fig. 1 .
Fig. 1.Traditional OOP and data-oriented ECS design for CPPS simulation.(a) Inheritance and abstract interfaces create complex object relationships to represent physical objects with IED under the OOP paradigm.(b) Entity is defined by data component combinations similar to a row in a data table under the ECS framework, while data are processed by columns.

Fig. 2 .
Fig.2.Real-world IED consists of the controller, communications, IO modules, and power supply.

Fig. 3 .
Fig. 3. IED sampling, control, and communication systems execution in the proposed data-oriented framework of ECS-Grid.

Fig. 4 .
Fig. 4. Example plugins and their configurations in the proposed ECS-Grid.

Fig. 7 .
Fig. 7. System topology and the detailed configuration of the 711-node microgrid cluster.
), while MMC-1 was impacted by the disconnecting event at t = 15 s.Fig.10(e) shows the real power balancing between three MMCs, where MMC-1 provides 2 MW and other MMCs drain the planned 1 MW from the dc system; the reactive power was set to 0 MVar; however, it seems to have large deviation at MMC-1 which may cause the larger voltage fluctuation in MG-1.The results are verified against theoretical analysis and commercial PSCAD/EMTDC; all data are measured from vIEDs with the interval of 100 ms and recorded by remote self-made supervisory control and data acquisition (SCADA) system.The results of Scenario 2 are shown in Fig.10(g)-(i).The dashed line indicates the waveforms under cyberattack while the solid line indicates the normal reactions; since the frequency of MG-1 was 50.02Hz after islanded, the operator sent a command at t = 147 s to reset P ref to 0.8 since the current real power is 0.8 p.u., which is a secondary frequency regulation process to restore rated operation point.Fig.10(g) shows the frequency of VSCDroop_0 in MG-1; under cyberattack situation, the frequency regulation command was intercepted and replaced to F ref = 0.5p.u., which generates very drastic deviations in all measurements from IEDs after t = 148 s such as the voltages

Fig. 10 .
Fig. 10.Simulation results.Scenario 1: Islanding operation.(a) Frequency of three Droop_0 stations in each microgrid.(b) Real power of each Droop_0 station.(c) Bus Voltages of each Droop_0 station.(d) MMC DC voltages during the islanding operation.(e) Real power of each MMC station.(f) Reactive power of each MMC station.Scenario 2: Simulated cyberattack.(g) Frequency comparison between normal operation and cyberattack situation of Droop_0 in MG-1.(h) Droop_0 Bus voltages comparison.(i) Droop_0 EMT bus voltage waveforms captured by virtual fault recorder.

TABLE I VIED
LATENCY AND BANDWIDTH USING EPROSIMA FAST DDS (RTPS)

TABLE II VIED
LATENCY AND BANDWIDTH USING ECLIPSE PAHO MQTT

TABLE IV VIED
POINT-TO-POINT LATENCY AND BANDWIDTH USING ZEROMQ (TCP)

TABLE V
VIED LATENCY AND BANDWIDTH USING ZEROMQ (TCP WITH BROKER)

TABLE VI FTRT
PERFORMANCE WITH VARIOUS COMMUNICATION INTERVALS