Approach for Identification of Geometry Change on Cultural Heritage Surface

Application of optical metrology techniques in the collection of surface data and its 3D representation can improve the digital documentation of the conservation and restoration process of artworks. The tracking of induced change after the restoration process on cultural heritage (CH) surfaces involves a computational analysis of surface geometry. In the analysis, the conservation scientists were interested to see what impact the fillings of some holes have on its nearby surrounding during the reconstruction. In theory, the loss compensation method for stone should allow conservators to make a filling that only exists in the place of the void, but it is highly unlikely to make a filling that will only adhere to the substrate at the void site and not protrude elsewhere. According to the conservator scientists, we proposed an approach of local geometry changes to identify and visualize changes and presented the outcome through a local neighborhood distance histogram. This analysis will give us overall surface change considering each surface point and its respective neighborhood points and what impact it faced due to the reconstruction process. The work is also focused on developing the representation of each type of loss compensation method to make it more objective according to a restorer ’s point of view and simplify their work visibility.


Introduction
The computational tracking of change on cultural heritage objects is important overtime to pass the information to the future [1].For the CH surface analysis and identify the change in this work, we considered a sculpture of a 'Lying Lion' shown in Figure 1.The sculpture of the 'Lying Lion' is one of two statues situated in an antithetical order on the foreground of the main entrance of the Radziwiłł Palace in Nieborów.It was sculpted probably by Johann Georg Plersch's workshop in the 18 th century.The figures of lions were not designed for the Palace in Niebrów.They were purchased after the first world war from the Tyszkiewicz family estate in Landwarów, [2] however, how they got there before and what was their original purpose remains unknown.
The statue is made as a freestanding sculpture its overall dimensions are 150 cm long, 71 cm wide and 86 cm high.It was made from a single sandstone block.Light cream, medium-grain sandstone composed mainly of quartz, with a significant amount of pyrites and siliceous-clay cement and characteristic visible sedimentation layers with rusty and yellow shades.The stone material resembles the Kunowski sandstone both visually and chemically.
The lion is shown in a reclining position, with its head elevated, facing the right side, the front paw is slightly tucked towards the body, the right one stretched forward, the rear right paw is stretched along the body, while the left is tucked under the belly.The sculpture is not anatomically correct, the lion's head is disproportionately large concerning the rest of the body.The whole composition is static and majestic, together with the twin antithetical figure, it rhythmically highlights the architectural axis of the palace.The sculpture surface has been carved in a variety of ways, the smoothness of the body is contrasted with the intensely wavy, rhythmically sculpted mane and rich texture of the base imitating hard stone blows, juxtaposed with softly modeled floral elements.
The overall condition of the statue before conservation was poor.The surface was deteriorated, with many missing parts, holes, and cavities.Some of them are impact damages and the loss of material was due to a mechanical impact, that most likely took place during multiple transportations of the statue.It was assumed that due to the movement of the monument also caused the splintering and chipping in the lower parts of the stone base.Another type of change that could be observed on the surface is small and larger holes created by the loss of iron-rich component, that together with bursting were caused by the corrosion and expansion of a large number of pyrites, that are sometimes naturally contained in this type of sandstone.Other forms of weathering that appeared on the surface were networks of thin cracks (craquelure), vertical and horizontal hair cracks, delamination, and exfoliation that are characteristic of laminar stones.As a result of air pollution and stone decay, the statue was covered with grey and black layers of weathering crusts, of different thicknesses and hardness, with local streaks and stains that harmed the visual reception of the carving.Therefore, both the structural integrity of the stone and aesthetic value of the monument were disrupted and needed to be restored by implementing the necessary conservation and restoration work.The focused area for our analysis (face) is shown in Figure 2 before the restorer fill the holes.The first stage of work on the lion figure was to describe and photograph the sculpture to document its condition.Afterward, the 'Lying Lion' was dismantled from its base and transported to the workshop of the Faculty of Conservation and Restoration of Works of Art, at the Academy of Fine Arts in Warsaw.Works on this object were carried out as part of the course classes of students of the 3rd and 4th year in the 'Studio of Conservation and Restoration of Sculptures from Sedimentary and Crystalline Rocks'.
The stone was disinfected, then samples for analysis in the laboratory were taken, and tests were carried out to determine the characteristic of the stone and weathering crusts and composition of the fillings and protective coatings from previous renovation interventions.Locally disintegrated and deformed lime mortar and cement mortar restorations from previous repairs, which did not match the original, were gently removed.The sculpture was precleaned with water under pressure adjusted to the strength of the stone and the hardness and cohesion of its weathering crust.During the treatment, a hydrophobic effect of uneven intensity was noticed in some parts, probably a result of the use of hydrophobic coatings during some previous conservators' intervention.
Based on the laboratory analysis results, attempts were made to remove the false patinas layers.Due to the relatively low presence of water-soluble salts in the stone, uneven water absorption of the surface and a high risk of migration of iron oxides that could result in appearance of more rust-brown stains, it was decided to abandon chemical methods of loosening compounds of the weathering crust and replaced them with cleaning by laser ablation.
At this stage of conservation, the 3D scanning was performed of a fragment of the lion's head.
Then the stone was saturated with solvent-free stone strengthener on a silicic acidethyl ester base KSE 300 and seasoned.The statue was covered with a Lotexan N colourless, silane/siloxane based, water-repellent.The scope of restoring numerous missing parts in the sculptural form, and the surface was set to a minimum extent for preservation purposes, and to enable the correct visual perception of the composition.The loss compensation methods were performed in two ways: as inserts of replacement stone, and as composite repairs -'artificial stone' presented in Figure 3. First method was to attach inserts made of stone, which have satisfactory aesthetic qualities and properties that are most likely to be acceptably compatible with the original stone.The second way was to create an "artificial stone" made of a material that can be well formed for some time, and then hardens in place, adhering tightly to the substrate and filling the void of the loss.
Due to the presence of hydrophobic residues the use of supplementary masses based on mineral binders turned out to be impossible.Selected for this monument were mineral fillers matched to the technical and visual properties of the original stone and Leposil K-1, a two-component silicone binder for putties.
The first type of reconstruction was made in places where the loss of material is extensive and massive.In the area of the lion's mouth, it was both eyebrows.Prior to stone carving, models were molded from plasticine.Plaster casts were made to recreate the break edges on the reconstructed parts to facilitate the alignment of stone inserts.The joint surfaces were covered with a thin barrier coating of Paraloid B-72 acrylic resin, for better reversibilty, then glued with epoxy resin Akepox 5010.After bonding the joint was slightly undercut to a depth of approx.5 mm, to use a supplementary mass for a better visual appearance.
The smaller gasps were filled with a mixture of synthetic binder Leposil K1 with stone fillers of various fractions (several types of crushed sandstone, quartz and marble powder, cristobalite and pigments.
In theory, the loss compensation method for stone [2] should allow conservators to make a filling that only exists in the place of the void, but it is highly unlikely to make a filling that will only adhere to the substrate at the void site and not protrude elsewhere.For this work, the restorers were interested to see the local geometry changes impact on the surroundings, where our analysis provides more objectivity and simplify the representation restoration work and improve the digital documentation.In Figure 4 the object's focus area after filling of the holes is presented.According to the conservator scientists needs, we proposed an approach of local geometry changes to identify and visualize changes and presented the outcome through a local neighborhood distance histogram.The behavior of a particularly interesting point on the surface up to a selected area will help to analyze and visualize the surface geometry change computationally.The work is also focused on developing the representation of each type of loss compensation method to make it more objective according to a restorer point of view and simplify their work visibility.

State of the Art
In the state of the art, for geometry analysis of cultural heritage surfaces the study showed analysis of change on global basis.Also segmentation was done to recognize geometry structure on one phase of data where no change detection was grouped.Some of the recent approaches for geometry analysis is described here in this section.

Change Analayis
Dante Abate et al. in [3], carried out an analysis to detect changes i.e deterioration of cultural heritage surfaces to improve the conservation method.An intuitive tool was generated for the production of degradation maps, and the identification of areas which have been affected by chemical and physical changes over time.Several threshold values were set to detect 3D change assessment coupled with close range photogrammetry.The drwabacks of the analysis was it had a need of fixed size of data from before and after conservation.The alignment was done only for a fixed scaled data from each phase.
In [4], Robert Sitnik et al. carried out a 3D monitoring of CH surfaces overtime and the analysis showed a colormap visualization of the degradation and detachment of parts from the surface based on eucledian distance calculation.The results presented was on global geometry change overtime.The drawback of the analysis was use of physical marker on the surface for a reference for the alignment.
F. Peteler et al. in [5], tried to quantify the detected 3D change using an image base modelling of ortho images.Several degradation types were performed in four different steps, creating manually the loss and the adding of material within intervals of a few days on the surface.The limitation to the analysis was resampling of the data each time for each computation.

Segmentation
In [6], P. Theologou et al. presented a fully automatic mesh segmentation scheme using heterogeneous graphs.They proposed some geometric measures to evaluate the segments produced by each eigenvector to avoid dependency on non-informative eigenvectors.Although the method was not robust to local noise or naturally not smooth regions.The segmentation was not based on surface points and not for two phases of data to group changes on surface.
Robert Sitnik et al. in [7], presented a fully automated segmentation, but the study was not for surface change grouping from monitoring.The segmentation was on one phase of data with limitation of definition of several parameters which characterized the input data based on the number of points, density, size of the measured object and noise.
In [8], L. Tchapmi et al. showed an end-to-end framework to obtain 3D point-level segmentation that combines the advantages of neural networks, tri-linear interpolation and fully connected Conditional Random Fields.However, the analysis showed the results are more accurate in large scale data set, in smaller data set the method needs to resample the data by a random augmentation or rotation or transformation or sub sampling.

Methodology
In this work, we decided to take information from the object's surface (Face) both before and after the restoration process to analyze and visualize the induced changes on it.For this work, we collected the surface data with a device which is a 3D scanner based on a structured light method developed at the Faculty of Mechatronics, Warsaw University of Technology (WUT) having a resolution of 0.25mm [9,10].The post-processing of the collected surface information was done in FRAMES which is written in C++ developed software at the Mechatronics Faculty of WUT.

Reconstruction 3D Models
The data collected from each phase are filtered up from the possible thermal noise of the scaner and the external environment considering a suitable threshold value from Hausdroff distance calculation [4,11].The noise free data are stitched together in the same coordinate space using a manual alignment and introduced to iterative closest point (ICP) algorithm [12] with the possible minimum root mean square (RMS) error for both phases of the 3D model.

Cross Time Alignment of the 3D Models
The cross time alignment of two models without a physical marker is indeed tricky and chance of getting error in the calcultation is more likely.In [4] it was solved using physical reference on the surface.To solve the cross time alignment of both the models in this analysis, we decided to make the ICP algorithm change sensitive where the weight of points is lowered for the changed areas and kept the same for points on the no-change areas.For the second phase of data i.e. after restoration into "no change" and "changed" only "no change" points were used for ICP but the global transformation was applied to all after restoration point cloud.

Assessment of Global Geometry Change
To assess the overall global geometry induced during the restoration process we calculated the nearest point to point distance from the before phase to the after-phase data.After the calculation of global change, we set the calculated distance values for each point to the before phase data set to present the obtained results with a colormap visualization.The global geometry change calculation was not enough to present the restorer's work as shown IS&T International Symposium on Electronic Imaging 2021 3D Imaging and Applications 2021 064-3 in Figure 6.Therefore, the approach of this work was focused on analyzing the change induced in terms of local geometry analysis.

Calculation of the Noise from the Datasets
To make the analysis insensitive to the resolution of the data captured by a particular scanner here we incorporated a calculated noise in unit of mm.We calculated average-point-to-point distance for both before and after phase point cloud and consider mean of both in this analysis.
This analysis considered (≃ 1-10% ) random points from total point count and considered the neighboring points for the selected points.Here, we set a parameter scaling factor as user input so the consideration of the neighboring points upto a radius would be multiplication of the scaling factor and the average point to point distance.From the considered neighboring points we fitted a best fitting plane on the points as shown in Figure 7. Then the distance between the random points and the best fitted plane was considered to calculate the RMS value of the distances for each random points.Then the RMS values were sorted in ascending order and skipped the lowest and highest values upto a percentage (≃ 5%) given by the user.From the remaining RMS values the average we considered as noise for our analysis.

Assessment of Local Geometry Change
In this work, the optimized searched algorithm of sphericalneighborhood search [13] is considered to find the nearest points around each point from the before restoration surface.For the spherical search the radius of the sphere was considered as average point to point distance of the data multiplied by a co efficient factor as user input.The consideredred neighboring points were also searched and analyzed to the after surface points creating a local distribution of the geometry.The behavior of the change concerning the local distribution of the neighborhood points based on Euclidian distance to analyze the impact of changes on the local-geometry of the surface as shown in Figure 8.Based on the obtained histogram for each surface points and its distribution curve behavior the analysis could visualize the reconstruction types of change as described in Figure 3

Classification of the Obtained Changes
In this section, the local geometry histogram was assessed and classified the similar behavior of the surface points, and group them.From the visual inspection of the local geometry change behavior, it was confirmed that for our work the data obtained are normally distributed.The behavior of the histogram was analyzed using the Kernel Density Estimation [14] function and the smoothing function for each data set.The bandwidth of the data distribution was calculated following the recommendation [15] of Silverman (1986), DATA PLOT which uses a default width of 0.9 * min(s, IQR/1.34)* n − 1/5 (1) Here, s is the sample standard deviation and IQR is the sample interquartile range.
The optimal width can be calculated as where n is the number of points in the raw data and s is the sample standard deviation of the raw data.
After the iteration of each surface point for its considered neighborhood points' distance, the bandwidth of the data set is stored for further verification.The bandwidth behavior is inspected and based on that we tried to classify different types of changes induced on the surface during the restoration process.For this work we could identify the mentioned behavior of surface as shown in Table 1 and respective local geometry histogram behavior and parameters behavior as shown in Table 2. Based on the Table 1 and Table 2 the types of filling specified by the restorers was identified and visualized.

Visualization of the Detected Changes
Easy representation of the change classification and grouping with color map visualization was considered to the work to make it a more user-friendly application for the common users.Each detected type of change was shown on the object's surface with specified color to make the restorer's work computationally visible for digital documentation.In our analysis we presented a No-Change type with color blue, the filling as cyan and replacement of artificial stone type filling as yellow.

Results Analysis
The analysis was computed and performed in FRAMES which written in C++ and designed developed at the Faculty of Mechatronics Warsaw University of Poland.The computation for the data is much faster compared to other platform.On the same platform the results were visualized and verified by the restorers.The results obtained obtained with a scaling factor of 2 for local neighborhood selection and a threshold of 15mm is shown in Figure 9 on the face of the lion.

Conclusion and Discussion
This analysis gave us both global and local geometry change considering each surface point and its respective neighborhood points and what impact it faced due to the reconstruction process.The analysis could quantify the global geometry change of 0mm to 49.23mm which was estimated as 0-40 mm by the restorers.The approximation given by the restorer was 0-10mm of addition of material on the composite filling and 20-40mm for the replacement of artificial stone.In the analysis, the quantified change of the composite filling needed a 15mm of threshold which clarified that the composite filloing varied from 0-15mm of addition of material and beyond 15mm it considered as linear filling of material on the replacement area as presented in yellow color in Figure 3.
The results obtained for the analysis were verified by a group of the restorers to justify the usefulness of the proposed analysis and visualization for the respective changes.The novelty of this work lies in the change-based analysis of two phases of data and having access to each surface point's change information.This work will help to analyze the local geometry change based on its local distance histogram behavior, which will provide more objectivity to the restorers' work.The analysis will also improve the 3D documentation of the restoration process with the quantification of added material for each type of fillings.

Challenges and Future Perspective of the Analysis
In the analysis, the major challenge faced was stitching of the scans to have the 3D models from each phase and cross alignment of the two phases of the 3D models without using a physical marker.This work is purely focused on change analysis, therefore to avoid the registration error and gain more accuracy in the analysis the density of the point cloud from the change parts was lowered, focusing alignment to the area where filling was done and copied the global transformation of no change parts from before restoration to the after restoration data.
In future note, this work is still on process, where we are trying to get more histogram behavior of changes.Also the analysis will be carried and tested on various CH surfaces to claim IS&T International Symposium on Electronic Imaging 2021 3D Imaging and Applications 2021 064-5 the strength of the analysis.A keen challenge is also getting access to the CH objects to test this analysis.Further analysis will be done from the cross sections of the each detected change types to get more detailed information from those area behavior.For this work, we will also consider multi modal data from various imaging techniques and make the method more automated and compatible for different CH surfaces.

Figure 1 .
Figure 1.The chosen object, right side view before conservation (Photograph by Agnieszka Leszkowicz, Barbara Wabnik, Przemysław Wysk, under the supervision of Roman Stasiuk)

Figure 2 .
Figure 2. The focused area before filling (Photograph by Sunita Saha)

Figure 3 :
Figure 3: The loss compensation methods: Yellow: insertion of replacement stone, Red: composite repairs (Photograph by Anna Duda-Maczuga)

Figure 5 :
Figure 5: The 3D models (Face) obtained from before(a) and after(b) the restoration process

Figure 6 :
Figure 6: Results obtained from global geometry change analysis

Figure 7 :
Figure 7: The noise calculation representation

Figure 8 :
Figure 8: The focus point (a) yellow and obtained spherical neighborhood points and its local geometry behavior (b)

Table 1 :
Identified surface behavior and local name for the change

Figure 9 :
Figure 9: Results obtained from local geometry change analysis shown on the lion face