Augmented Reality in Tracking the Surface Geometry Change of Cultural Heritage Objects

Augmented reality is a field of presenting and visualizing the real-world problem enhanced by computer-generated algorithms. The computer-generated perceptual information becomes a premier research topic to solve various 3D problems. In this present age, it is very important to digitize the process of work and store it to keep track of the changes over time. This work is focused on storing digital documentation of the restoration of cultural heritage objects concerning the parameter of occurrences of surface geometry change on the surface of the objects. The paper shows the application of structured light techniques to the analysis of a cultural heritage object during restoration. In order to detect and visualize changes a global geometry analysis was applied and presented the outcome through a global distance histogram. The study also includes the discussion of facing possible challenges during the development of the algorithms for the change analysis.


I. INTRODUCTION
The conservation and restoration of cultural heritage objects is an activity of preserving the material witnesses of human art [1]. The matter of work of art is an essential way of transmitting aesthetic and spiritual content.
The analysis of the states of preservation [2] of the cultural heritage object is a starting point for conservation, however not the only one. Monitoring the changes during the aging process and restoration treatment is indispensable for responsible care of monuments and works of art. For many years both chemical and physical analysis is an important part of conservator work. They could not be properly used without close cooperation with a scientist from various fields.
Augmented reality is an efficient tool for documentation and analyzing several processes in the field of preserving cultural heritage objects [3]. 3D imaging techniques are useful in every case of documentation when there is only need to analyze surface geometry [4]. It means that sculptures or reliefs, any painting surfaces, textiles, different objects representing ethnographic art and of course architecture and many others are the fields of implementation of these technics. 978-1-7281-6644-5/20/$31.00 ©2020 EU One group of problems is monitoring the aging of the objects as aging processes are usually much extended in time. The very important for the preservation process is to capture any changes at the very early stage. Using those methods it is possible to get an overall picture of changes not only on macro but also on micro-scale. This enables conservators to plan treatment at the stage when there are still no irremediable changes in the monument. It is also a unique tool to verify storage conditions. The second area of application of 3D imaging techniques is the ongoing monitoring of conservation processes. Analysis of the effectiveness and accuracy of methods of removing layers of deposits is the most obvious application. Nevertheless, the accuracy of today's measurement techniques also allows us to capture small changes, sometimes invisible, that may indicate the appearance of stress inside the object as a result of direct conservation work or interference in the close surroundings.
Working during restoration treatment on cultural heritage objects are now supported with very advanced technology and imaging systems [5]. The object conservators, by undertaking their practical activities in the conservation process, are creating the reception of works of art by their contemporaries. Their actions are ultimately conditioned by human sensitivity and cultural references. That is why the cooperation of scientists able to create augmented reality with those who ultimately touch with the human hand, and thus not the perfect tool, historic monuments are so important. Mutual understanding and work experience in these two different areas aimed at protecting cultural heritage are fundamental. The issues presented in this paper are just one of these research programs combining the perspective of technologically advanced documentation with work with the restorers of heritage objects. Restoration of artworks is enhanced by digital documentation that allows documenting pre and posting restoration conservation status. In this paper, we tried to present some preliminary developed methods and tried to test the methods on mock-ups for validation of the proposed method. The goal of the work is in line with the application of optical metrology techniques to cultural heritage questions.

II. 3D IMAGING TECHNOLOGY
There are various techniques to digitize [6] the cultural heritage objects and the researchers are continuously trying to develop the process of dealing with different phases of the process. 3D imaging is a great scientific tool to work in the automated and interactive data processing. The development of the visualization method can be beneficial in conservation processes in monitoring the surface to detect and assess the object before and after changes. This work is enhanced complete with 3D digitization of a selected object for each stage of the restoration process. It is important to have the 3D information from each phase of the process to analyze the changes computationally.
Digitization of cultural heritage is the process of complete digital recording [7] of the surface points of the cultural heritage objects. This process depends on the nature of the object and the reason of its digitization. The entire process of recording involves the selection of the objects to be digitized, method of collecting the information from the object, digital data processing, analyzing, visualization and reconstruction of the surfaces. The 3D digitization [8] system is influenced by the suitability of application of a method on an object such as: 6. The complexity of the developed algorithm in terms of space and time.
In the last decade, the conservation scientists accepted to adopt 3D imaging technology to document digital copies of the interesting artworks appreciating 3D representation.
The 3D imaging is a technology of capturing 2D images of a particular object from different viewpoints and then analyzing the images to gather the information from the object's surface on 3D space. 3D scanning is based on photogrammetry which stitches the 3D data to get the real view based on 2D images. The complete 3D imaging is a whole set of work packages of capturing the data, removing the noise from the data, reconstruction of the complete 3D model and then post-processing of the obtained data. 3D imaging is important to the preliminary documentation of artworks to produce a complete image without sampling and any analytical computation.
There are various techniques and thousands of research is going on to develop the 3D imaging technology [9] both in terms of image capturing set up and based on computational analysis of the captured images. It is considered as one of the high-performance scientific tools for examining and analyzing artworks. The capture system for the 3D imaging is always prepared with the focus on quality of the output. In this case the measuring device is focused on high-resolution 2D images with the minimum possible low-cost components those are used to prepare the measuring device.

III. METHODOLOGY
The computational analysis cultural heritage field is a challenging task in terms of its availability and accessibility and restriction to work on it. The change tracking in cultural heritage objects is a time-consuming process that requires a lot of time if natural aging is considered, practice and accuracy assurance to test the method on real objects. In this section, to test the proposed method we have chosen two mockups to work on it as shown in Fig. 1 and Fig. 2  After considering the mockups for the examination, in this work, we chose the structured light (SL) method for scanning the objects. In this work, we tried to calculate the occurrence of the change in terms of surface geometry or texture of the objects. We collected the surface points with a custom-designed structured light-based device as shown in Fig. 3.
In SL scanner images are captured using projected light patterns and camera systems [10] where the projector is treated as an active camera. The projection of a narrow beam of light onto the object produces a line of illumination that appears distorted from other perspectives than that of the projector, and can be used for geometric reconstruction [11] of the surface shape for the object.
In this 3D scanner, to capture the acquisition data from a chosen object's surface, many parallel stripes or arbitrary light fringes are projected at once and the same is continued from different viewpoints to collect the complete surface data from the object. The displacement of the light fringes on the surface of the object during scanning allows for an exact retrieval of all the details on the object's surface in 3D space. The geometric distortions [12] of light and perspective are remunerated by a calibration of the data collection device, using the selective calibration patterns and surfaces. The detailed information about the scanner components used in the device is listed in Table 1. The projector and camera are mounted on joints that allow justifying their position relative to each other. The joints allow changing the position of the camera / projector and fixing it permanently. The laser ruler is also mounted on a rotating head with a blockade. The rotary table for positioning the measured detail (also used during calibration) is located at an appropriate distance from both scanners; this distance is coupled with the focal distance of the lens used in the camera and projector. The measurement volume for the used structured light scanner is 140 x 280 x 180 mm with an approximate resolution (the average distance between points in a point cloud) of 0.2mm.
After obtaining data of the surface geometry from both the selected mockups, certain changes were introduced to both the objects as shown in Fig. 4 and Fig. 5. Since the work is to detect the change in an object in terms of surface geometry computationally, mockups were introduced a change in shape to analyze the change in this case.

IV. RECONSTRUCTION
The stitching of the point clouds was performed in Frames which is developed in Warsaw University of Technology using C++. To get the complete 3D shape for the objects from the obtained scans reconstruction was performed based on the efficiency of the shape reconstruction algorithm [13,14]. During the scanning process, some points appeared from some discontinuous parts on the surface of the object due to the nature of the 3D scanner. In this analysis, the point groups which appeared at a distance from the main point cloud based on some given threshold value were considered as noise. The resulting noise was removed based on following a threshold value of Hausdorff distance criterion.
After the filtration of the noisy points in each of the scans the filtered point cloud were stitched together to get the complete 3D representation of the mockups. The registration of the point clouds was done based on the manual shifting of the clouds and then introducing it to iterative closest point (ICP) algorithm [15]. The reconstruction obtained from the scans of the mockups before making any change is shown in Fig. 6 and Fig. 7 for both the objects.
In this work, the 3D representation of the selected mockups was obtained by removing the challenge of putting a physical marker on the object's surface. The global registration of each scan obtained during the scanning was tried to fit together into its best fitting plane only based on ICP algorithm. The same procedure of reconstruction was followed to obtain the 3D representation for both the mockups after introducing the certain changes on its surface as mentioned in section II of the paper. In Fig. 8 and Fig. 9 the 3D model is shown after the changes made on the surface of the mockups. All the reconstructions presented in this work were done with the minimum possible radius value of the ICP algorithm to find the closest point between each scans. The analysis to track and visualize the changes made on the surface of both the mockups was done using CloudCompare v2.11 alpha software based on the parameter of Euclidean distance between the point clouds after putting each reconstruction in the best fitting plane. This test analysis is following the work of [16], however, in this work; we tried to avoid user-defined physical markers to fit in the same plane. The preliminary results obtained for the global distance analysis for the before and after changes made for both the prepared mock-ups are shown in this section. Also based on the obtained results a visualization map is presented for the known changes on the surface of the objects.

VI. CONCLUSION AND DISCUSSION
The obtained results were fair enough to justify the changes made on the objects and to understand the resulting behavior based on the material of the object, color of the object and also the working principle of the measuring device. The registration of each scan without any physical markers as a reference is also a challenging task which may lead to inaccurate results. The obtained result as presented above having an inaccurate distance value on some parts of the object where no changes were made was considered one of the major drawbacks of the registration phase. The fitting of each scan has to be on the best-fit plane to get more accuracy in results. An easy solution for this registration phase could be of defining a reference point on the surface of the object, but in this work the goal is to avoid a physical marker. The registration of point clouds is an open problem in each field of application of 3D documentation and representation. This analysis is strongly dependent on this phase of 3D representation.
The challenges in combining both the conservation and scientific imaging and also in terms of accessibility of cultural heritage object to work on, this analysis was worth in having an idea of the change tracking process. The investigation in terms of tracking a change occurs in cultural heritage objects requires a good study of the artifact that needed to be examined. The examination reason for a specific object and examination techniques concerning the object chosen have to be discussed with a specialist i.e. a conservation scientist in case of cultural heritage first to check their availability to proceed with the examination with a chosen artifact.
This analysis was done to get familiar with the change tracking methods and to obtain a realistic view of the problems and also before applying on any kind of cultural heritage objects. Real-world is a good bunch of problems in terms of 3D data representation computationally. It is a good practice to work on samples or mockups before going to a real application. However, it's difficult to estimate how far the acquisition complexity working on the real objects will be the same scenario as presented in this work concerning various circumstances faced by the objects. The analysis may show different results depending on the material, aging, faced humidity, certain climate change, chemical reaction, lighting condition, rainfall, location, etc. on real objects.
In addition to detecting and asses, the changes on the surface of the CH object this work will also try to measure the value of the change occurrence on the object. The ideal goal of the work is measuring the exact value of the change based on local neighborhood analysis for an interesting point on the surface of the object. Based on the analysis of the change in a particular point our motive is to define the change according to the histogram behavior of the change analysis on the surface of the object.

ACKNOWLEDGMENT
This work is carried out at the beneficiary partner of CHANGE: Cultural Heritage Analysis for New Generation, Warsaw University of Technology, Poland, supported and funded by the European Union's Horizon 2020 research and innovation program under the Marie Skłodowska-Curie grant agreement No. 813789.