Ant Colony Optimization Algorithms for Dynamic Optimization: A Case Study of the Dynamic Travelling Salesperson Problem

nt colony optimization is a swarm intelligence metaheuristic inspired by the foraging behavior of some ant species. Ant colony optimization has been successfully applied to challenging optimization problems. This article investigates existing ant colony optimization algorithms specifically designed for combinatorial optimization problems with a dynamic environment. The investigated algorithms are classified into two frameworks: evaporation-based and population-based. A case study of using these algorithms to solve the dynamic traveling salesperson problem is de - scribed. Experiments are systematically conducted using a proposed dynamic benchmark framework to analyze the effect of important ant colony optimization features on numerous test cases. Different performance measures are used to evaluate the adaptation capabilities of the investigated algorithms, indicating which features are the most important when designing ant colony optimization algorithms in dynamic environments.


I. Introduction
Ant colony optimization (ACO) is a well-known metaheuristic for combinatorial optimization problems inspired by the foraging behavior of real ant colonies [1], [2].Ants are able to find the shortest path between a food source and their nest [3].They lay pheromones (a chemical substance produced by ants) on the ground to mark their path, forming in this way a pheromone trail, and they tend to follow paths marked by strong pheromone concentrations.In this way, the ants are able to share information with their nest mates.
In ACO, a colony of (artificial) ants cooperates in constructing high-quality solutions to difficult combinatorial optimization problems [4].The construction of solutions is guided by (artificial) pheromone trails.The pheromone model used to update the pheromone trails is inspired by the pheromone trail laying and following behavior of real ants.This is basically a parametrized probabilistic model that is modified by the ants to reflect their experience while optimizing a particular problem.
ACO was initially designed for solving static combinatorial optimization problems [4].However, the environments of many real-world problems are often dynamic [5].A dynamic combinatorial optimization problem can be seen as a series of different static problem instances.Hence, a straightforward method to deal with these type of problems is to consider each dynamic change as the arrival of a new problem instance that needs to be solved from scratch.However, this method is often impractical when the dynamic change is relatively small [5]- [7].A better strategy is to adapt to dynamic changes by transferring the past experience of the optimization process since the new environment will be in some sense related to the old one.ACO is a good choice in adapting to dynamic changes because it naturally implements a memory structure (i.e., the pheromone model), allowing ACO to remember the past experience.Therefore, when a dynamic change occurs the past experience can be transferred via the pheromone trails of previously optimized environments [7].Successful ACO applications to dynamic combinatorial optimization problems include Internet-like network routing [8], vehicle routing [9], and train scheduling [10].
In the last decade, studying ACO algorithms for dynamic environments attracted a lot of attention because of their intrinsic features [6], [7].Several dynamic strategies have been designed to enhance the adaptation capabilities of ACO [11]- [14].These strategies have been mainly tested on different dynamic variations of the traveling salesperson problem (TSP), which makes the dynamic TSP (DTSP) an ideal subject for a case study in this article.However, in the existing DTSPs [15]- [19], important dynamic optimization settings (e.g., the type, frequency, and magnitude of the dynamic change) are not following any experimentation protocol.This causes many difficulties for researchers to analyze the strengths and weaknesses of algorithms in DTSPs.Consequently, the following important questions arise: Which ACO algorithm performs best under which dynamic settings and why?Which ACO features are important when addressing dynamic environments?We strongly believe that it would be beneficial to have a unified dynamic benchmark framework to evaluate algorithms in DTSPs with common dynamic settings [20], [21].
This article aims to provide insights concerning the behavior of ACO using the DTSP as a case study.In order to achieve this aim, we set out the following two objectives.First, the existing ACO algorithms designed for DTSPs are classified according to their framework either as evaporation-based or as population-based.Second, a unified dynamic benchmark framework is proposed to systematically carry out a critical evaluation of the most important features of ACO algorithms (i.e., the decision rule for constructing solutions, the policy for updating pheromone trails, and the dynamic strategy for enhancing adaptation capabilities) on different test cases.In fact, this benchmark framework will also be able to serve as an initial proving ground for new algorithmic ideas in dynamic environments.
The rest of this article is organized as follows.Section II describes the generation of dynamic test cases utilizing the unified dynamic benchmark framework with the TSP as the base problem.Section III classifies the ACO algorithms from the literature that have been designed for DTSPs.Section IV outlines the experimental setup of our study.Next, Section V presents and analyzes the experimental results.Finally, concluding remarks are presented in Section VI.

II. Dynamic Travelling Salesperson Problem
In this section, we describe how dynamic test cases can be generated from a static problem instance.TSP is used as the base problem to generate the dynamic test cases in this article because it is a problem without too many technicalities (e.g., hard constraints).Hence, it is more convenient to evaluate algorithms because their behaviors will not be obscured by the technicalities of the problem [22].Additionally, the TSP is an important N P -hard combinatorial optimization problem arising in several applications [23].Finally, the best ACO algorithms for the TSP very often perform well when applied to more complex problems.For example, the Ant Colony System [1], which is one of the best performing ACO algorithms on the TSP, is used to solve world-scale instances of vehicle routing problems [24].

A. TSP Formulation
The TSP can be described as follows: given a collection of cities, the objective of a salesperson is to find the shortest Hamiltonian cycle of visiting all of the cities once before finally returning to the starting city.More formally, a TSP problem instance is modeled by a complete directed weighted graph ( , ), G N A = where N is a set of n nodes and A is a set of arcs fully connecting the nodes.For the classical TSP, nodes and arcs represent respectively the cities and the links between the cities.Each arc ( , ) i j A ! is associated with a non-negative value , w R ij !+ which represents the distance between nodes i and .
j In this article, we will use symmetric TSP problem instances to generate dynamic test cases and, hence, these distances are independent of the direction of traversing the arcs, that is, w w ij ji = for every pair of nodes.

B. Generating Dynamic Test Environments
Every TSP problem instance consists of a weight matrix that contains all the weights associated with the arcs of the corresponding graph .G In order to generate dynamic test cases the weight matrix of the problem is subject to changes as follows: where ( ) W $ is the weight matrix and T is the environmental period index which is synchronized with the algorithm during the optimization process.Therefore, the environmental period index is defined as / , T t f = ^h where f is the frequency of change and t is the evaluation counter of the algorithm.
A particular TSP solution r = [ , , ] is a permutation of node indices, and for the DTSP it is evaluated as follows: Mainly, there are two components of the graph G representing the problem that can change: 1) the set of nodes [13], and 2) the weights on the arcs [11].A change to any of these problem components will also cause a change to the weight matrix defined in Equation 1and, thus, it may affect the algorithm's output: the best output before a change may not be the best (or even feasible) after the change.
Real-world applications that encompass the aforementioned types of dynamic changes can be found in many fields, including transportation.For example, changes in the traffic situation (i.e., weight changes) or changes in the visiting locations (i.e., node changes).

1) DTSP with Node Changes
The  , ]) m 0 1 !defines the magnitude of change.The higher the value of , m the more nodes will be replaced.In this way, the weight matrix will be affected because the weights on the arcs connecting the replaced nodes will be modified.

2) DTSP with Weight Changes
A dynamic test case with this type of changes can be generated by assigning an increasing/decreasing factor value to the arc connecting nodes i and j as follows: where ( ) w 0 ij is the initial weight of arc ( , ) i j (from the static TSP instance when ), T 0 = Rij is a normally distributed random number (with zero mean and standard deviation set proportional to ( ) w 0 ij as in [20]) that defines the modified factor value to arc ( , ), i j and ( ) A T A S 1 defines the set of arcs randomly selected for the change at that period.Consider that the size of the set A is defined by the number of arcs as follows: ( ). n n 1 -Then, the size of ( ) A T S is defined by the magnitude of change (i.e., ( , ]) m 0 1 !and the size of .A Therefore, every f evaluations exactly ( ) mn n 1 -^h arcs will be selected to change their weights.The higher the value of , m the more arcs will be selected for changes.

C. Some Additional Remarks
It must be noted that there are some ACO algorithms that benefit from the use of less ants (e.g., the Ant Colony System [1]), resulting in less evaluations per algorithmic iteration, and some other ACO algorithms that benefit from the use of more ants (e.g., the hyper populated ant colonies [25], [26]), resulting in more evaluations per algorithmic iteration.Therefore, the frequency of change is expressed in evaluations in the described dynamic benchmark framework.In this way, a fair comparison between the competing algorithms is ensured with the generated DTSP test cases because the dynamic changes occur at exactly the same period of time (i.e., all algorithms have exactly the same number of evaluations available between the environmental changes).Also, the period of change is restricted either at the start or at the end of an algorithmic iteration [27].
From the way the magnitude of change is defined for both DTSP types, there exists an interesting observation: the weights of the same number of arcs, but not necessarily the same arcs, are modified when the same value of m is selected for DTSPs that utilize symmetric problem instances.For example, in Figure 1 the dynamic change with magnitude set to .m 0 2 = will change one node for a problem of size five (i.e., .
) 0 2 5 # ^h when node changes occur and four arcs for the same problem (i.e., . ( ) ) 0 2 5 5 1 # -^h when weight changes occur.Consider that there are two arcs connecting a pair of nodes and their weights are the same in symmetric problem instances.For a problem of size five in a fully connected graph, each node is connected with the remaining four nodes; therefore, when one node is replaced, the weights of eight arcs in total will be modified as shown in Figure 1(a).For the same problem, the weights of the four selected arcs will be modified when weight changes occur, but, due to the symmetric property of the problem instance, each time the weight of an arc ( , ) i j changes, the weight of the arc ( , ) j i changes to the same value.As a result, the weights of eight arcs in total will actually change (as in the case of node changes) as shown in Figure 1(b).

A. ACO for the Dynamic Travelling Salesperson Problem
In the ACO metaheuristic, a colony of ~ artificial ants constructs solutions by incrementally selecting feasible solution components.Each solution component is associated with a pheromone value which is used to guide artificial ants when selecting the next solution component.Since the TSP is considered as the base problem to generate dynamic test cases, it is used as a concrete example to describe the ACO metaheuristic in this section.
With probability q 1 0 -( [ , ]) q 0 1 0 !each ant chooses the next node probabilistically.The probability with which ant k chooses node j from node i is defined as follows: , , , , while with probability q0 each ant k chooses the next node j with the highest probability as follows: where ij x and / ( ) are, respectively, the pheromone trail value (which is initialized with a value ) 0 x and the heuristic value (which is available a priori) of the arc connecting node i to node , j a and b are two parameters that control the relative influence of the pheromone versus the heuristic information, and N i k is the set of nodes that ant k has not selected yet when being at node .i When ., q 0 0 0 = we have the normal random proportional decision rule in which ants make moves probabilistically using only Equation 4. When ., q 0 0 0 2 we have the so-called pseudorandom proportional decision rule, introduced in the Ant Colony System [1], in which ants make the best possible moves as indicated by the existing pheromone trails and the heuristic information using Equation 5, in combination with the probabilistic moves defined in Equation 4.

B. ACO Framework Types
There are two framework types of the ACO metaheuristic.First, the evaporation-based framework that typically utilizes the random proportional decision rule as shown in Algorithm 1.Second, the population-based framework that typically utilizes the pseudorandom proportional decision rule as shown in Algorithm 2. These two frameworks also differ in the way their pheromone trails are updated and, consequently, in the way they adapt to dynamic changes.

1) Adapting via Pheromone Evaporation
The evaporation-based framework used in this article adopts the pheromone update policy of the -MAX MIN Ant System ( ) AS MM [28], which is one of the best performing evapora-tion-based algorithms.The pheromone trails are updated by applying evaporation as follows: where ( ( , ]) 0 1 is the evaporation rate.After evaporation, the best ant deposits pheromone on the arcs belonging to its constructed solution (i.e., ) x uniformly with a value 0 x 3: while termination condition is not satisfied do 4: construct ~ solutions using the random proportional rule in Equation 45: evaluate all solutions using Equation 26: values are always updated proportionally to the solution quality of the current best-so-far ant.
When a dynamic change occurs, the current pheromone trails will contain mixed information that can either guide the search towards promising areas of the search space containing high-quality solutions or misguide the search towards poor areas of the search space containing low-quality solutions.Hence, some previously generated pheromone trails may lead to a possibly poor solution for the new environment.Therefore, pheromone evaporation is responsible for the gradual reduction of these pheromone trails.

2) Adapting via Population-list
The population-based framework uses a population-list ( ) P I of size K (i.e., an archive of K solutions), in which at each iteration I the solution of the iteration-best ant is stored in the population-list [12].The population-list is directly associated with the pheromone trails, that is, whenever an ant in enters the population-list, a positive pheromone update is performed as follows: where Tx is the constant pheromone amount to be deposited and in r is the solution of the ant entering the population-list (i.e., ).
ib in r r = Whenever an ant out leaves the population-list, due to the limited size of ( ), P I a negative pheromone update is performed as follows: where Tx is the constant pheromone amount to be removed and out r is the solution of the ant leaving the population-list (i.e., the oldest solution in the population-list).In this way, outdated pheromone trails are reduced directly when a dynamic change occurs.
There exists an interesting relationship between the evaporation-based and population-based frameworks: they both use pheromone trail limits, which is an important feature to avoid early search stagnation [29], although these are explicit, as discussed earlier, in the evaporation-based framework and implicit in the population-based framework.In particular, the pheromone trail values in the population-based framework can never drop below the initial pheromone value 0 x because a constant amount of pheromone Tx will always be added to the trails when the ant enters the populationlist and the same constant amount of pheromone Tx will always be removed from the same trails when the ant leaves the population-list.On the contrary, the additional pheromone value a trail can receive can be at most , K which is the size of the population-list, times the constant amount of pheromone .Tx Hence, in the population-based framework is implicitly guaranteed that ( , ): .
+ It must be also noted that the pheromone trail limits (i.e., min 0 = + in the populationbased framework are fixed, whereas the pheromone trail limits in the evaporation-based framework are variable (they change whenever a new best solution is discovered).

C. ACO Variants for the DTSP
As discussed earlier, both ACO frameworks can successfully address DTSPs using their pheromone trails to utilize previous experience and adapt to the newly generated environments, either via pheromone evaporation in the case of the evaporation-based framework (denoted as AS MM in Table I) or via the population-list in the case of the populationbased framework (denoted as P-ACO in Table II).However, these adaptation mechanisms often become ineffective once ACO shows a stagnation behavior (i.e., when all the ants follow the same path and construct the same solution over and over again).This is because excessive growth of pheromone trails will be generated on the arcs of a single solution.Therefore, in case a dynamic change occurs, it will be difficult for the ants to escape from this solution (which could be optimal previously) in order to search for the new optimal solution.Since both frameworks focus their search on a specific area of the search space, there is a consequent danger of getting trapped in a stagnation situation.Although the pheromone trail limit feature, discussed in Section III-B, is designed to counteract this situation, both ACO frameworks need to be further endowed with additional features when addressing dynamic environments [14].Several dynamic strategies have been designed to balance the knowledge transferred and the diversity maintained in ACO, and, consequently, improve the adaptation capabilities of ACO in dynamic environments [7].Achieving a good balance between these two factors is not a trivial task.On one hand, increasing the diversity of the constructed solutions resolves the issue of search stagnation.On the other hand, it may disturb the optimization process because of too much randomization.Additionally, transferring knowledge is essential for faster recovery when dynamic changes occur.However, if too much knowledge transferred or utilized from previous environments, then the reoptimization process may start near a (possibly poor-quality) local optimum and get stuck there.In fact, these two factors are also conflicting because if the diversity maintained is not enough, then it will be difficult for ACO to utilize any knowledge transferred.
Existing ACO variants for the DTSP are classified into evaporation-based and population-based framework variants in Tables I and II, respectively.The main common feature of all these ACO variants is that they prevent the concentration of large amounts of pheromone trails on the arcs of a single solution.In the evaporation-based variants, this is typically achieved by reducing the pheromone trails on these arcs much faster than the remaining pheromone trails and/or increasing some pheromone trails on arcs of other promising solutions in the search space.On the contrary, the population-based variants typically introduce newly generated solutions, that do not belong in the constructing colony, in the population-list.In this way, the population-list is unlikely to maintain identical solutions that will result in stagnation of the search.It must be noted that the dynamic strategies of most variants are based on explicit actions when a change occurs.For detailed descriptions of these variants, refer to their original references listed in Tables I and II.

A. Dynamic Test Cases
TSP instances were obtained from the TSPLIB benchmark library [40], which is available at http://comopt.ifi.uni-heidelberg.de/software/TSPLIB95/,to generate dynamic test cases as described in Section II.Specifically, the frequency of change was set proportionally to the size of the problem instance as follows: .
indicating quickly (e.g., before the algorithm converges, denoted as fast) and slowly (e.g., after the algorithm has converged, denoted as slow) changing environments, respectively.The magnitude of change was set to ., m 0 1 = ., m 0 25 = ., m 0 5 = and ., m 0 5 7 = indicating small, to medium, to large dynamic changes, respectively.The dynamic settings for each DTSP test case are selected to systematically analyze the dynamic behavior of ACO algorithms (i.e., their ability to recover fast and produce the best output).Note that usually as the frequency of change is faster and the magnitude of change is increasing the DTSP test case becomes harder to address [14], [21].

B. Parameter Settings
The common parameters of all ACO algorithms were set to typical values as follows [4]: = for ACO algor ithms using these parameters.
Note that these parameters were found to yield reasonable performance in most DTSPs.In general, the best results on both types of DTSPs are obtained when setting the evaporationbased variants with a high pheromone evaporation rate and the populationbased variants with a small populationlist size.On the contrary, the colony size varies for the two types of DTSPs: a smaller colony size performs better in DTSPs with node changes, whereas a larger colony size performs better in DTSPs with weight changes.
For each ACO algorithm on each DTSP test case, 30 independent runs were executed on the same set of random seed numbers.For each run, 100 environmental changes were allowed and the value of the best-so-far ant since the last change of the environment was recorded.

C. Performance Measures
Five metrics for evaluating the performance of ACO were considered: 1) offline performance, 2) best before change, 3) robustness, 4) diversity, and 5)m branching factor.The first two metrics are classified as optimality-based metrics, whereas the last three metrics are classified as behavior-based metrics.The set of metrics is described in the following: ❏ Offline performance ( ) Poffline r measures how well an algorithm performs until it finds a high-quality solution [5].It is one of the most frequently used metrics for dynamic optimization, which is defined as the average solution quality of the bestso-far solution found at each evaluation step.This metric is calculated as follows: ( , ), where ( , ) is the quality of the best-so-far solution since the last change of the environment and E is the total number of evaluations (which is defined as , E Mf = where M is the total number of environmental changes and f is the frequency of change).❏ Best before change ( ) Pchange r measures how good the final outcome of an algorithm is [41].It is another useful metric, defined as the average solution quality of the best-so-far solution found at the end of each environmental period.This metric is calculated as follows: is the quality of the best-so-far solution achieved just before the T-th environmental change.❏ Robustness ) (Probust r measures the stability of an algorithm when dynamic changes occur [32], [42].Suppose that at evaluation count t a change occurs.Then, the robustness of an algorithm is calculated by comparing the quality of the best-so-far solution just before the change (i.e., at evaluation count ) t 1 with the quality of the best-so-far solution when a change occurs (i.e., at evaluation count ) t as follows: , .
The values of Probust r are normalized and range over the interval [0, 1], where a value closer to 1 indicates better average robustness because the quality of the best-so-far solution will have less degradation at each environmental change.❏ Diversity ( ) D r gives an indication of the amount of exploration an algorithm performs [32], which, in this case study, is calculated as follows: where ( , ) CA p q is the total number of common arcs between the solutions constructed by the p-th and q-th ants.The D r value tends to decrease as the ants start to follow similar paths, where a value of zero corresponds to stagnation of the search.❏ m -Branching factor ( ) m r measures the distribution of the pheromone trail values [43].The m-branching factor is given by the number of arcs incident to node i satisfying the following Finally, in order to support our comparisons, pairwise Wilcoxon rank-sum statistical tests with a significance level of 0.05 were performed.In the case of multiple comparisons, Kruskal-Wallis statistical tests were performed, followed by posthoc pairwise comparisons using Wilcoxon rank-sum statistical tests with p-values adjusted by Bonferroni correction.

V. Experimental Results and Their Analysis
This section presents representative examples of the results obtained in our experimental studies and analyzes them 1 .

A. Comparison between Evaporation-Based and Population-Based Frameworks
In this study, the evaporation-based framework is compared with the population-based framework.The experiments were carried out using three TSPLIB instances to generate the dynamic test cases: kroA200, rd400 and u1060, in which the number in the instance name identifies the problem size, and they are considered as small, medium and large instances, respectively.Table III shows the , Poffline r , Pchange r and Probust r results obtained by the two frameworks for DTSPs with . .m 0 25 = To better understand their behavior, Figure 2 shows plots of the m r (with .) 0 05 m = and D r results against the last ten node changes on the TSPLIB instance rd400.From the experimental results, the following observations can be drawn.The detailed results of each experimental study are provided as supplementary material.The supplementary material is available at https://github.com/Mavrovouniotis/ACODTSP.
First, in terms of , Poffline r the population-based framework performs significantly better than the evaporation-based framework in most quickly changing DTSPs.This is because the pseudorandom proportional decision rule used in the population-based framework is more greedy than the random proportional decision rule used in the evapora-tion-based framework and, thus, it directs the search towards high-quality solutions quickly.On the contrary, the random proportional decision rule requires sufficient time to explore the search space (and most probably discover better-quality solutions).Hence, the evaporation-based framework performs significantly better, in terms of both Poffline r and , Pchange r than the populationbased framework in most slowly changing DTSPs.Another reason is that the evaporation-based framework requires some time to make significant changes to the pheromone trails, because the pheromone evaporation will gradually decrease outdated pheromone trails down to the explicit min   population-based framework decreases outdated pheromone trails much faster, because the pheromone on the arcs belonging to the solutions to be removed from the population-list will be directly set to the 0 x value (which is also the implicit min x value).Nevertheless, it appears that the time required for the random proportional decision rule to express its effect is affected by the scale of the problem.For example, on the TSPLIB instance u1060, the population-based framework significantly outperforms the evaporation-based framework even in slowly changing DTSPs.This is natural because in large problem instances with huge search spaces the random proportional decision rule will spend an enormous amount of exploration time until it discovers areas in the search space containing high-quality solutions.
Second, the population-based framework usually maintains a higher m branching factor than the evaporation-based framework [see Figure 2(b)].This is because the pheromone trails of the evaporation-based framework are more likely to represent a single solution (i.e., the solution of the best ant).In particular, the pheromone on the arcs corresponding to this solution will rise up to the explicit max x value, while on all the other arcs the pheromone will decrease down to the explicit min x value.On the contrary, the pheromone trails of the population-based framework, are more likely to represent multiple solutions (i.e., the solutions stored in the population-list).Hence, the pheromone on the arcs corresponding to these solutions will increase (and most probably have the same value), while on all the other arcs the pheromone will remain equal to the implicit min x value.Third, in terms of , Probust r the population-based framework significantly outperforms the other framework in almost all DTSPs.Nevertheless, the robustness of both frameworks is very high for DTSPs with weight changes.This is because the environmental changes generated by the weight changes may not affect the current best-so-far solution (the arcs in which their weights change may not necessarily belong to that solution).On the contrary, if a dynamic change occurs to the nodes, it will always affect the current best-so-far solution and, consequently, affect the current optimization process.Recall from Figure 1 that all arcs incident to the affected node will be modified.Therefore, it is guaranteed that at least one affected arc will belong to the current best-so-far solution.
Fourth, it is interesting to observe that although the averagem branching factor of the population-based framework suggests more exploration than the evaporation-based framework [see Figure 2(b)], the average diversity the former framework generates is usually lower than the latter framework [see Figure 2(a)].This is because the pseudorandom proportional decision rule exploits the search experience accumulated by the colony more strongly than the random proportional decision rule as discussed previously.Consequently, the solutions constructed will have a relatively large number of common arcs, resulting in lower D r results.

B. Effect of Main Framework Features
In this study, the two main features of the evaporation-based and population-based frameworks are investigated: 1) decision rule (i.e., random denoted as R or pseudorandom denoted as P), and 2) pheromone update policy (i.e., proportional denoted as P or constant denoted as C).Specifically, the two ACO frameworks with their default decision rule and pheromone update policy are compared with alternative feature combinations (in the format "ACO framework"_"decision rule"_"pheromone policy").For example, the default feature combinations of the evaporation-based and population-based frameworks are denoted as Evaporation_R_P and Population_P_C, respectively.The experiments were carried out using the same set of TSPLIB instances as in the previous study.Figure 3 shows the Poffline r results obtained by the two frameworks with all possible feature combinations on the TSPLIB instance kroA200.
From Figure 3(a) it can be observed that the constant amount of pheromone has no effect on the performance of the evaporation-based framework.This is because only a single ant is allowed to add pheromones in each iteration.Also, from Figure 3(b) it can be observed that the proportional amount of pheromone degrades the performance of the population-based framework.This is because the resulting amount of pheromone representing the population-list solutions is very small to bias the search.
Furthermore, the use of the random proportional decision rule has a negative effect on the performance of the population-based framework.As we have seen previously, the way pheromones are distributed on the arcs of the constructing graph promote search space exploration.This is reflected by a higherm branching factor for the populationbased framework [see Figure 2(b)].Therefore, a decision rule that further promotes the exploration may result in too much randomization.Similarly, the use of the pseudorandom proportional decision rule has a negative effect on the performance of the evaporation-based framework.This shows that the two main ACO framework features often complement each other.

C. Effect of Dynamic Strategies
In this study, the dynamic strategies integrated with the evaporation-based and population-based frameworks are investigated.The experiments were carried out using a wider set of TSPLIB instances than in the previous studies.From the statistical comparisons of the evaporation-based and population-based frameworks against their variants, listed in Tables I and II, respectively, the following observations can be highlighted.
If a dynamic change occurs to the nodes, it will always affect the current best-so-far solution and, consequently, affect the current optimization process.
First, evaporation-based variants performing severe explicit actions, such as AS MM R and Max-Strategy variants, are not effective for DTSPs with weight changes.As discussed earlier, the environmental changes generated by weight changes usually affect the current bestso-far solution less than when the environmental changes are generated by node changes.Hence, a reinitialization of the pheromone trails (either to the initial pheromone value with AS MM R or to the maximum pheromone value with Max-Strategy) may destroy useful previous knowledge.On the contrary, these variants perform significantly better than the evaporation-based framework in most DTSPs with node changes (except when ., m 0 1 = in which the changing environments are more likely to be similar).
Second, MC-AS MM and AS MM caste variants perform significantly better than the evaporation-based framework in several DTSPs with node changes, while they maintain a competitive performance in DTSPs with weight changes.These variants focus their search on multiple areas of the search space (either using multiple colonies with MC-AS MM or multiple castes with ). AS MM caste Therefore, the past experience available to be utilized is broadened when a change occurs.
Third, EIACO, MIACO and HIA-CO-II variants perform significantly better than the population-based framework in most DTSPs, both with node and weight changes.It is interesting to note that these variants have a common feature: their dynamic strategy generates elitism-based immigrants.This shows that this dynamic strategy is responsible for the performance improvement.Elitism-based immigrants are generated via transferring knowledge from the previous environment and replace solutions in the population-list to maintain diversity.Therefore, when the changing environments are similar (e.g., when .), m 0 1 = the utilization of this knowledge will guide the search process to promising areas faster.
Fourth, RIACO and EIIACO variants are effective only for a few extreme DTSP test cases (i.e., when f is fast and .). m 0 75 = Otherwise, they perform significantly worse than the population-based framework in most DTSPs.This is due to the fact that the diversity generated is very high and therefore the ongoing optimization process is disturbed.

D. Effect of Utilizing Change-Related Information
From a practical perspective, changerelated information may be available in real time to the optimizer (e.g., using advances in information and communication technologies or related technologies, in the case of transportation systems).In this study, the outdated solutions are repaired using changerelated information: the affected nodes are removed and the newly introduced nodes are placed in the best possible position when a change occurs [12].The experiments were carried out on the same set of TSPLIB instances as in the previous study.Figure 4 shows the Poffline r results obtained by the evaporation-based and population-based frameworks when change-related information is utilized and when it is not, on the TSPLIB instance kroA150.In addition, these results are compared with the results obtained by the R e s t a r t -S t r a t e g y, - h Strategy, and x -Strategy variants that were specifically designed to utilize changerelated information as listed in Table I.
From F ig u re 4, the following observations can be drawn.First, the performance of both ACO frameworks  is significantly improved when changerelated information is utilized in all DTSPs.Second, the three competing strategies outperform the two ACO frameworks only when they do not utilize change-related information.This shows the effectiveness of utilizing such kind of infor mation to explore and/or exploit the affected areas in the search space faster.

E. Comparisons with Evolutionary Algorithms
In this study, the two ACO frameworks (i.e., AS MM and P-ACO) and two of the best performing variants (i.e., MC-

MM
and EIACO) are compared with a state-of-the-art evolutionary algorithm that utilizes one of the best performing TSP search operators (i.e., the generalized partitioned crossover (GPX) [44]) and the elitism-based immigrant genetic algorithm (EIGA), which is one of the best performing algorithms in evolutionary dynamic optimization [20], [45], and also the evolutionary counterpart of EIACO.
Since we compare algorithms which are structurally different, the frequency of change is set to f n 100 = (which is considered as slow) to allow sufficient evaluations between the dynamic changes for all types of algorithms.The magnitude of change is randomly chosen from a uniform distribution in ( ., .] 0 0 0 5 (which is considered small to medium) for every dynamic change.The experiments were carried out on a different set of TSPLIB instances from the previous studies.Table IV shows the Poffline r results obtained by the aforementioned algorithms for DTSPs with weight changes.
From Table IV, it can be observed that the two ACO frameworks outperform the evolutionary algorithms in almost all DTSPs.This is because for this class of problems (i.e., with a network environment) the pheromone structure of ACO algorithms is built across the solution search space as a weighted graph and it can be taken as the natural representation of past environmental information.Therefore, there is a larger capacity of information that ACO can utilize from its pheromone structure when a dynamic change occurs in comparison to evolutionary algorithms (which are restricted only to the information of their evolving populations).
Also, it is interesting to observe that EIACO outperforms EIGA although both algorithms utilize the same dynamic strategy.Once again, it can be observed that: 1) EIACO outperforms P-ACO in most DTSPs, 2) MC-AS MM is competitive with , AS MM and 3) P-ACO outperforms AS MM as the size of the problem increases.These comparisons are consistent with the observations found in our previous studies.

VI. Conclusions
This article investigates the behavior of the ACO metaheuristic in dynamic environments.Several ACO algorithms are classified according to their framework (i.e., evaporation-based and population-based).The traveling salesperson problem is used as the base problem to generate dynamic test cases using a proposed dynamic benchmark framework.Experimental studies were systematically conducted to investigate the effect of different features on the performance of ACO in dynamic environments.
From the experimental results, the following concluding remarks can be  drawn.First, the effect on the performance of the decision rule used to construct solutions strongly depends on the pheromone update policy of the ACO framework.Second, the dynamic strategies further enhance the adaptation capabilities of the two ACO frameworks, but their effect on the performance strongly depends on the dynamic settings of the problem.And third, the utilization of change-related information is always effective for both ACO frameworks.
Finally, the source code of the dynamic benchmark framework together with the performance measures and the ACO algorithms used in this case study is available at https://github.com/Mavrovouniotis/ACODTSP.The source code can be useful to researchers who are interested in generating the same or different dynamic test cases to compare their own algorithms (not necessarily ACO) in dynamic environments.

FIGURE 1
FIGURE 1 Illustration of the weight matrix when node (a) and weight (b) dynamic changes (with .m 0 2 = in this example) occur.Gray boxes denote a change to the weight.Note that the symmetry with respect to the main diagonal line is due to the fact that symmetric problem instances are used to generate DTSPs.

! and t 0 ! 2 :x 3 :! 4 4: 5 6 8 : 21 :Algorithm 1
initialize all pheromone trails ij x uniformly with a value 0 set the population-list ( ) P I while termination condition is not satisfied do 5: construct ~ solutions using the pseudorandom proportional rule in Equations 4 and find the best solution b i r from the I-th iteration 9: if population-list ( ) P I is full then 10: select the oldest solution out r to leave the population-list ( ) OUTPUT: the best-so-far solution bs r Evaporation-Based Framework.1: I 0 !and t 0 !2: initialize all pheromone trails ij

1 a = and 5 b 5 ~=
= (for all the experiments).The colony size ~ for DTSPs with node changes was set to

n 2 1 -
. The average m -branching factor (i.e., ) m r from all nodes' m -branching factors gives an indication of the level of search space exploration generated by the ants.The values of m r range over the interval[ ,  ], where a value of 2 indicates stagnation behavior.

FIGURE 2 D
FIGURE 2 D r (a) and m r (b) (averaged over 30 runs) results of evaporation-based and population-based frameworks against the last ten environmental changes.

FIGURE 3
FIGURE 3 Poffline r (averaged over 30 runs) results of evaporation-based (a) and population-based (b) frameworks with alternative decision rules and pheromone update policies.*These combinations are the default ones.

FIGURE 4
FIGURE 4 Poffline r(averaged over 30 runs) results of evaporation-based and population-based frameworks, and three evaporation-based variants.Each bar is divided into two parts that represent the results when utilizing change-related information (darker) or not (lighter).*These variants have been designed to utilize change-related information and, thus, the values when information is not utilized do not exist.

TABLE I
Dynamic strategies of evaporation-based variants.

TABLE II
Dynamic strategies of population-based variants.

TABLE III
Bold values indicate statistical significance.

TABLE IV
Poffline r results (averaged over 30 runs) of ACO algorithms compared with evolutionary algorithms for DTSPs with weight changes.
Bold values indicate statistical significance.Underline values indicate no statistical difference with the bold value.