Multiagent System for Community Energy Management

: Local energy communities (LECs) represent a shift in energy management from an individual approach towards a collective one. LECs can reduce energy costs for end-users and contribute to meeting climate objec-tives through the use of renewable energy. This paper presents the application of a multiagent system (MAS) approach to realize the concept of LEC in a real-world scenario involving a community of households. An appropriate agent-based model for the given community is presented. This model effectively distributes the tasks among the agents considering electrical and heat energy ﬂows. The agent coordination mechanism is based on the Alternative Direction Method of Multipliers. The obtained results provide evidence of the validity of the developed MAS and show its potential to increase a total social welfare of the community.


INTRODUCTION
Local energy communities are considered a promising way to integrate distributed energy resources and to engage end-users in sustainable energy practices. In LECs, participants can locally buy and sell their energy. When production inside the community is not sufficient to meet local demand, the electricity shortage is covered by import from the main grid. On the other hand, the surplus energy can be exported to the grid. By acting together, community members have a stronger negotiation power when interacting with other energy market participants. The trade inside the community is encouraged by the difference between local and retail prices. Trading locally is beneficial because it allows funds remain within local economy. This also reduces losses that occur when the energy is transmitted over long distances.
The recent interest in LECs contributed to a growing number of industrial projects and research publications (Sousa et al., 2019). Some focus on the design of a peer-to-peer (P2P) market with necessary functions where peers can buy and sell their energy. Thus, (Mihaylov et al., 2014) proposed a virtual currency to regulate energy exchange between peers. The payment functions for those supplying and consuming energy were developed to encourage a https://orcid.org/0000-0002-6847-8313 b https://orcid.org/0000-0003-1140-4433 c https://orcid.org/0000-0001-8646-6183 the energy balance. The approach is dependent on function settings (Lilliu et al., 2019) and does not account for a strategic behavior of peers. The latter is often addressed using a game theory. (Paola et al., 2017) proposed price-based schemes and a gametheoretical framework to coordinate flexible demand. Though, distributed generation and energy storage systems were not taken into consideration. (Shamsi et al., 2016) suggested an auction-based market mechanism, where each household provides bids or offers of their demand or generation. These offers and bids are collected to allocate energy and determine prices. In the auction based market, accurate estimates of energy and prices are needed to get better tariffs. This can be disadvantageous to inexperienced participants. A major advantage of P2P schemes is that they are able to preserve privacy as there is no central supervisory entity.
A more structured market design enables a cost sharing mechanism where community members pay in the form of a share of a single electricity bill of the overall community. This design assumes the coordination of actions towards the common goal. Thus, (Long et al., 2018) proposed a two stage aggregated control to realize a P2P energy sharing where an energy sharing coordinator controls flexible devices. An interior-point method was used to minimize the energy costs of the community. A centralized in nature is its major disadvantage that is also common to many existing approaches based on energy sharing. The centralization limits scalability and rises concerns about privacy. Multiagent systems (MAS) are promising alternative to decentralize computations. (Kantamneni et al., 2015) surveyed MAS applications for microgrid control. Energy trading in the community using MAS was recently addressed by (Vinyals et al., 2018). Although this approach presented an agent-based model of the community, a limited number of flexible devices and only the flow of electricity were considered.
This study presents a multiagent system developed to manage a community of households located in the central Netherlands. Our approach is close to cost sharing ones. We describe the developed agentbased model of the community energy network. This model accounts for the flows of electrical and heat energy. The model conveniently decomposes the underlying optimization problem so that each agent is assigned with a specific task that can be solved efficiently. In particular, we present an agent-based decomposition for addressing the model of battery with losses. This contrasts with existing approaches. For example, (Yang and Nehorai, 2014) addressed the problem directly introducing additional parameter for optimization. Furthermore, the presented MAS approach is not limited to the considered community and can be adapted to other real-world scenarios.

CASE STUDY
The present study has been carried out in the scope of a European project whose aim is to unlock a demand response potential in the distribution grid. The addressed case study involves a community of households located in the central Netherlands. Figure 1 depicts the architecture of community and aggregator platform. This figure presents a logical architecture rather than the physical one. This view is applicable to manage: (i) households located in a close neighborhood sharing a common point of connection with the main grid, as shown in Figure 1 and (ii) households that are geographically distant and have no physical common point coupling.
The considered community consists of 16 households and a district battery with the capacity of 220 kWh. Each house has a heat pump of 2 kW combined with a hot water buffer of 200L. The heat pump is used for heating both domestic hot water and spatial heating. Each house has solar photovoltaics (PV) capable of producing up to 7kW. Additionally, there is one in-home battery with the capacity of 7.8 kWh.
The given community is managed by an energy service providing company know as aggregator. The aggregator manages flexible assets in the community aiming to maximize the value of flexibility. The aggregator platform facilitates the interaction with external parties and provides an infrastructure for executing MAS. It relies on information and communications technology and consists of different communication devices, software applications and protocols. Each house is equipped with a local energy gateway (LEG) that is connected with flexible devices and is able to communicate with the backend run on a cloud. Thus, LEGs are used to control flexible devices and to communicate sensory data. The Advanced Message Queuing Protocol is used for communication.

MULTIAGENT SYSTEM
The community energy network is modeled as a multiagent system where agents are nodes and the environment is everything outside the agent. The state of the environment is represented by sensory data including the charge levels of batteries, indoor and outdoor temperatures, temperature limits for comfort, the prices of energy and possible limits for energy flows. The actions of agents are the amount of energy pulled from or injected to the grid.   Figure 2 shows the multiagent model of the community energy network. The energy network is represented by a bipartite graph having two set of nodes. This suggests two types of agents, shown by circles and rectangles. Circles depict device agents (D). The device can refer to an abstract or physical device. Rectangles show net agents (N). The net represents a virtual zone where the energy exchange between the devices takes place. Edges in the graph indicate interaction between agents. Table 1 lists different types of agents and explains their roles.

Multiagent Model
The model represents a structure of agent interactions. It extends a multiagent model of a building by aggregating different buildings and district battery using AGRNet. This net can be viewed as a local energy market where district actors negotiate their energy exchange. AGRNet is also connected to an external source of power that represents a connection point between the community and the main grid. Inside a building, there are electrical (shown in blue) and heating (shown in red) networks that account for the flows of electricity and heat. Net agents ensure the balance of corresponding energies. Connector agents that connect houses to the AGRNet represent prosumers in the local energy market. They communicate schedules for houses while keeping energy flows inside houses hidden at the community level. The battery with losses is represented by connector, net and battery agents. The agent that connects to the battery net models losses associated with charging and discharging. The battery agent models linear constraint associated with its capacity.
This problem decomposition has been performed to obtain an adequate representation of the energy network such that individual subproblems are easy to solve and yield meaningful solutions.

Optimization Problem
MAS addresses the problem of minimizing the energy bill at the connection point while satisfying energy balance constraints, physical constrains of devices and comfort preferences. Using a splitting technique for the variables associated with net and device agents the following multiagent optimization problem is formulated minimize where f i is a real valued objective function associated with the i-th device and defined in the feasible region Ω i , g i is a real valued objective function associated with the i-th net and defined in the feasible region Θ i , x i and z i are the decision variables associated with the i-th devices and net, respectively. The constraints account for the fact that respective device and net agents should agree upon the values of shared variables.
Finding the minimizer to an equality constrained optimization problem is equivalent to identifying the saddle point of the associated Lagrangian function. This gives the following augmented Lagrangian function (2) where u i = λ i /ρ is the scaled dual variable (for Lagrange multipliers λ T i ) , x i and z i are primal variables.

Agent Coordination
Agents solve their local problems in a coordination with their neighbors. The coordination mechanism is based on the alternating direction method of multipliers (ADMM). ADMM iteratively minimizes the augmented Lagrangian function (2) with respect to primal and dual variables. Each iteration involves the following steps.
Step 1. Device agents compute in parallel their optimal variables by solving The corresponding values are communicated to neighboring nets.
Step 2. Net agents compute in parallel their optimal variables by solving Step 3. Net agents in parallel update their dual variables. u The corresponding primal and dual variables are sent to neighboring devices. The above steps are repeated until convergence criteria are met. The convergence criteria are defined locally for primal and dual residuals as where ε primal , ε dual are small positive numbers representing primal and dual tolerances, respectively. The primal and dual residuals are computed as If the device and net functions f (x) and g(z) are convex, the constraint residual under ADMM is guaranteed to converge to zero and the objective value to the minimum of the dual problem, see (Boyd et al., 2011).

Nets
The net agents (AGRNet, ENet, HNet and BNet) ensure the energy balance treating the constraints The problem (4) is solved by projecting the variables received from neighboring devices onto the feasible region as in (Kraning et al., 2014).

Devices
The C agent connects two nets. In each time step τ, there two possible cases: (i) the energy flows from net 1 to net 2, then (3) is solved by finding critical points for two cases and selecting the one with a lower value of (3) in each time step τ. HP represents a particular case when the energy flows only from ENet to HNet.
The ET agent is associated with the following objective function where p buying (τ) and p selling (τ) are respectively the price of imported and exported energy, x + (τ) is the energy with the positive sign (imported), and x − (τ) is the energy with the negative sign (exported). The set of constraints restrict the energy coming from and into the grid.
where x min (τ) and x max (τ) are respectively the minimum and the maximum energy flow allowed. The problem (3) is solved by finding critical points for positive and negative cases and selecting the one with a lower value of (3). Constraints are treated by projection.
The FL and PV agents aim to satisfy respectively the forecast consumption and production The B agent has a set of constraints aiming to keep its state of charge as well as charging and discharging rates within the allowed range.
where Q min and Q max are the minimum and maximum allowed charge of the battery, x min and x max are limits for discharging and charging rates. The battery's charge evolves as (Kraning et al., 2014) where Q init is the initial charge of the battery.
The SH agent has a set of constraints to ensure a room temperature is within the comfort temperature limits.
where T min and T max are the temperature limits. The room temperature evolves as (Kraning et al., 2014) T where T (0) is the initial temperature, T amb is the outdoor temperature, µ is the conduction coefficient, η is the heating efficiency and c is the heat capacity of indoor air.
The HWS agent has a set of constraints to ensure the temperature of water inside the tank is within the allowed range.
T min ≤ T (τ) ≤ T max , τ = 1, . . . , H with T min and T max are the temperature limits. The water temperature evolves as (Tasdighi et al., 2014) T where T (0) is the initial temperature, V cold is the volume of water with temperature T cold entering the tank to replace the consumed hot water, V total is the tank volume, and c is the specific heat of water. The consumption of hot water is forecast. The B, SH and HWS agents solve the problem (3) using Dykstra's projection method with a starting point (z k − u k ).

Market situation Positive
Positive Overconsumption/Underproduction: the market is proposing the prosumer to consume the energy z i paying the price u i . Positive Negative Overconsumption/Underproduction: the market is proposing the prosumer to produce the energy z i and get paid the price u i . Negative Positive Underconsumption/Overproduction: the market is proposing the prosumer to consume the energy z i and get paid the price u i . Negative Negative Underconsumption/Overproduction: The market is proposing the prosumer to produce the energy z i paying the price u i .

Agent Negotiation
ADMM iteratively processes primal and dual variables of the augmented Lagrangian function. These variables have a meaningful interpretation within the energy network. The primal variables represent the energy and the dual variables define its price in different nodes across the network. Agent interactions, governed by ADMM, model a negotiation process where device agents negotiate their energy exchange in exchange zones represented by net agents. Messages sent from nets to devices represent requests. These messages contain the amount of energy requested, z k i . The sign is used to distinguish between production and consumption. From a recipient perspective, a positive value indicates that the energy flows towards the recipient, a negative value indicates the flow towards the sender. It also includes the price, u k i , which not only indicates the current market price, but also is used to distinguish between a situation of overconsumption (in which the price will be positive) and a situation of underconsumption (in which the price will be negative). Table 2 lists different market situations in exchange zones of energy network based on the sign of the price and the requested power. The first two cases (i.e. positive prices) refer to a market with underproduction (i.e. overconsumption) situation. The last two cases (i.e. negative prices) refer to a market with overproduction (i.e. underconsumption). In all cases, the suggested payment is given by u k i · z k i . The device agents send messages with offers as response to received requests. This establishes the precedence relation in agent communication meaning that offer messages are only sent after request messages have been received and processed. An offer message contains the amount of energy, x k i , offered by the device in each time interval.
The negotiation continues until the consensus is reached between the agents. The device agents agree on their energy profiles so that their variables are equal to those requested by the nets. This situation means the supply is equated to the demand in all the exchange zones and the corresponding markets are cleared.

RESULTS AND DISCUSSION
This section presents the results obtained by the developed MAS. The tests were performed using the data for the winter season. Winter days are characterized by the need to use heat pump for both heating domestic hot water and maintaining room temperature within comfort limits. This represents the most challenging optimization scenario as during other seasons heat pump is only used for reheating domestic hot water.

Optimizing Community Cost
Community members participate in community energy management having a common interest of increasing social welfare through collective use of local resources. This implies collaboration in terms of energy use to reduce the cost at the connection point when there is no external flexibility requests.

Day Ahead Optimization
Multiagent optimization suggests optimal energy usage for the next 24 hours discretized into 96 program time units (PTUs) corresponding to 15 minute time intervals. When running a day ahead optimization, the system attempts to schedule energy consumption for flexible loads to time intervals when renewable energy from solar panels is available. Starting optimization at different time slots during the day changes the perspective as the time horizon moves forward embracing new time slots from the next day. Figure 3 provides insights about the effect of executing optimization at different PTUs during the same day. The figure shows the energy profiles of the community at  the connection point with the main grid. Positive values refer to the amount of energy taken from the grid (imported energy). Negative values indicate the energy fed into the grid (exported energy). The plot on the left shows results when optimization was executed at midnight. The plot on the right refers to starting time at midday which embraces the data for the following day.
The battery allows storing the excess of renewable energy and later delivering this energy by discharging when local demand is high. This is why the community does not import energy during evening hours. The local demand is met by the energy released from the battery. MAS schedules the district battery charging and discharging so that it takes locally produced energy only in the amount needed to meet local demand until the end of the time horizon. Figure 4 provides much insight about this process. The plots show state of charge (SOC) with optimization starting at different PTUs. It was considered that in the beginning the battery had the lowest allowed SOC of 20%. During the day the SOC is increased by injecting the energy from solar PVs. Later this energy is released to supply local demand. No extra energy is left or released for export. Without additional constraints for optimization, this is the most effective strategy and is as expected. Because the use of battery is always associated with some costs due to charging and discharging efficiencies, the battery should only be used when that is really necessary.

Rolling Horizon Optimization
The method of applying optimization at every program time unit is known as rolling horizon optimization. Rolling horizon can mitigate uncertainties in the models, such as forecast errors, and account for the fact that the real time horizon is not limited to a single day. Figure 5 summarizes the results for rolling horizon optimization. The effect of rolling horizon optimization can be understood from Figure 5a. For comparison, this figure shows the community energy profiles obtained by day ahead and rolling horizon optimizations. The former provides the results from the single optimization run at the beginning of the day. The latter involves the results of 96 optimizations during the day, with the data for the first PTU being used for control. The difference can be readily understood. Rolling horizon optimization results in smaller energy exports during times of intense electricity generation from PV panels. This is explained by the fact that rolling horizon accounts for time slots in the next days with a need to meet local demand. The day for which results are presented in Figure 5a is characterized by a massive production from solar PVs panels. The optimized energy profiles suggest peaks in times of high PV generation. This is because no constraints on possible power flows were imposed. In such scenario, the system solely focuses on minimizing the energy bill of the community. Although the results indicate that rolling horizon optimization can reduce peaks to some extent, it is not the goal of the community at this step. The issue of dealing with possible grid overload is addressed in the next section as part of congestion management scenarios.
The optimal behavior of the community is characterized by shifting consumption of flexible devices to time intervals with electricity available from solar panels. This is because this electricity is cheaper than the one taken from the main grid. Thus, both in-home and district devices are expected to use energy in such times during the day.
Heat pumps are flexible devices at the home level. They are used to provide heat for both domestic hot water and space heating. As expected, MAS suggests using heat pumps in times with available renewable energy. This is shown in Figure 5b that depicts a three-dimensional bar chart with the consumption profiles of each heat pump in the community. It can be seen that most of consumption occurs in times of high PV generation. Consumption in other time intervals is dictated by the need to satisfy temperature limits, especially for domestic hot water. The consumption of domestic hot water is irregular with respect to time and amount, especially in the evening.
At the district level, there is a battery storage. When multiagent optimization runs in a rolling horizon mode, in the end of the day, the battery SOS is not at its lower state because it holds some energy to account for the following time horizon. This is shown in Figure 5c that summarizes 96 optimization runs for one day. Figure 5d graphically illustrates the control actions for the battery during the day in terms of the amount of energy charged and discharged in each PTU.

In-home vs Community Optimization
Community management should ensure its members benefit from collective energy usage. To provide insights about the advantages of community-based approach we compared the results of in-home and community optimizations. The former optimized community households individually and combined their energy profiles. The latter performed the community optimization. The results indicate the performance of two approaches varies depending on the amount of renewable energy that is locally produced. Communitybased approach is better when there is excess of renewable energy. On the other hand, both approaches perform similarly when it is low. Figure 6 illustrates the amount of imported energy for the two optimization approaches. It can be seen that for all days community optimization yields less or equal energy import as those of in-home optimization. Similar results are only observed when PV generation is low. The analysis of the results indicates that an increase in renewable energy generation leads to a decrease in energy imported from the grid. Table 3 further summarizes the obtained results. During the period under consideration, the total renewable energy production was 9047.91 kWh. Similarly to the above daily data, the total energy import and export of the community was reduced. It is important to note that besides economic benefits this also reduces the stress on the main grid. Instead of exporting the produced renewable energy it is used to meet local demand.
Self-consumption is an important performance indicator. It is defined as the ratio between the energy consumed and the total energy produced by the community. As shown in Table 3, the community based optimization approach increases selfconsumption. These results provide important insights about advantages of collective energy management and the estimates of potential savings. Figure 6: Comparing the energy import of the community when using in-home optimization individually and community optimization.

Congestion Management
Power volatility caused by renewables and irregular consumption poses significant challenges for the distribution grid. This volatility has the potential to lead to network congestion, decreasing reliability. Congestion management refers to avoiding the thermal overload of system components by reducing loads. Congestion occurs if the capacity of the grid is insufficient to accommodate the requested power flows. This section presents and discusses the results of addressing two different congestion management scenarios. In these scenarios, first the optimal community energy profiles were found by community optimization without considering grid capacity constraints. Next, it was considered that a grid safety analysis performed by a distribution system operator (DSO) required to limit the injected/consumed energy. The limits were introduced into the MAS in the form of constraints. The resulting constrained optimization problem was solved by multiagent optimization to reschedule the energy use in the community.

Overproduction
The first considered congestion management scenario refers to the situation in the grid with an excess of injected renewable energy. This can readily occur in sunny weather when local demand is low. Under such (a) maximum export 5kWh (b) maximum export 3kWh circumstances, an increase in voltage can happen and grid components can be damaged if no proper actions are taken. Figure 7 shows a day ahead scenario for the community addressing the oversupply of renewable energy. The energy profiles resulting from unconstrained community optimization are shown in blue. The results obtained when addressing flexibility requests from the DSO are shown in yellow. Two different requests limiting the energy injection are considered. These correspond to the maximum energy injection of 5 kWh and 3 kWh per PTU. It can be seen that both grid capacity limits are respected by the MAS. The amount of renewable energy injected into the main grid does not exceed the limits requested. This is achieved by shifting the export from peak sunny periods during the day to evening and night hours. This reduces the stress on the distribution grid and provides energy in times of high demand, such as evening hours.
The availability of the storage capacity in the community is critical to exhibiting the above behavior. The battery is able to take the excess of renewable energy in peak times and later release it. Since the community is interested in minimizing the cost and maximizing the profit, the losses in the battery should be minimized. Figure 8 shows the behavior of community storage in terms of energy charging and discharging in this scenario. This figure illustrates that the more limited grid capacity, the more energy is taken in peak times of PV production and correspondingly the more energy released later. This results are as expected and confirm the ability of MAS to adapt its energy export according to grid conditions.

Overconsumption
The second considered congestion management scenario involves the overload of grid components due to high demand. This situation is common in times with limited local production and high consumption. Figure 9 depicts the community energy profiles addressing these congestion management scenarios. The plots show the results for requests limiting peak consumption to 4 kWh and 3 kWh per PTU. For comparison, the plots also show results for community optimization without congestion constraints. Similarly to previously considered scenario, these results suggest that MAS is able to schedule the energy use for the community under considered congestion management scenarios.
The plots show that high demand occurs in evening hours resulting in a peak consumption around 16:00. The requests from the DSO ask for lowering this demand. To address this the community needs to anticipate peak demand and to use the energy available in off-peak times. The more power reduction is (a) maximum import 4kWh (b) maximum import 3kWh   requested the more energy is consumed during offpeak periods. This energy is taken by the community battery storage when the local demand is low. Later, during peak times, the stored energy is re-leased. Thus, the community does not overload the grid pulling energy in critical periods and meet its demand by using the locally stored energy. Figure 10 shows charging and discharging of community storage without and with congestion management. Without congestion, no energy is injected. This is because during the considered day there is no excess of renewable energy. All the energy generated by rooftop panels is consumed locally. However, when MAS takes into consideration DSO requests, the battery storage is utilized to charge in off-peak time intervals and discharge later to meet peak demand.
The reduction in peak demand can also be addressed by shifting consumption of flexible loads. Heat pumps are flexible loads in the considered community. Figure 11 shows the cumulative consumption of heat pumps in the community. A shift in the consumption from peak to off-peak periods can be observed. This shift takes place alongside with the use of battery storage. The energy consumed by heat pumps can come from the battery and from the exported energy as long as constraints are respected.
Thus, the obtained results show that flexibility requests from the DSO are met through efficient use of flexible loads, which combines charging and discharging of battery storage as well as shifting heat pump consumption to anticipate a peak load.

CONCLUSIONS
This paper presented a multiagent system for a realworld scenario of community energy management. An appropriate multiagent model was developed considering electrical and heating networks. The agent models were presented indicating solution methods for their subproblems.
The obtained results offered important insights and provided the evidence for the validity of multiagent system approach. Single run of multiagent optimization results in optimal control actions for all controllable devices in the community. These are in the form of consumed and/or injected energy as well as temperature set points for thermal loads. The system demonstrated the ability to offer solutions satisfying all the constraints whenever feasible solutions exist. These constraints involve allowed state of charge of battery, temperature limits in each time unit and energy balance in each node of the network. The suggested consumption profiles are meaningful. The energy consumption for flexible loads is scheduled in times when renewable energy is available. The export of energy is scheduled so that local demand is met first and only excess of local supply is fed into the main grid.
As opposed to many existing studies in the literature that consider artificial case studies where the number of prosumers and consumers are carefully selected, the presented study involved a real-world community of households. These households have similar characteristics and are located in a close neighborhood. As a result, rooftop solar panels produce renewable energy in same times during the day and in similar amounts. The consumption patterns of households are also similar, which is due to social and economic similarities of neighbors. Often, this leads to the situation when solar panels intensively produce electricity virtually all households have the energy in excess. In such times, the community has a surplus of cheap energy and no local demand to use it. As a result, an energy sharing mechanism is not fully appreciated by community members. One possible way to address it can be through considering more flexible loads. Another can be through encouraging households without PV panels to join the community. Such new community members would benefit from consuming the surplus of local renewable energy. Importantly, households that are the energy producers will also be better off due to differences in prices for export to the main grid and for selling to community peers.
As future work, we will address the issue of a fair distribution of costs and profits between community members.