A Survey of Machine Learning Techniques in Adversarial Image Forensics

Image forensic plays a crucial role in both criminal investigations (e.g., dissemination of fake images to spread racial hate or false narratives about specific ethnicity groups) and civil litigation (e.g., defamation). Increasingly, machine learning approaches are also utilized in image forensics. However, there are also a number of limitations and vulnerabilities associated with machine learning-based approaches, for example how to detect adversarial (image) examples, with real-world consequences (e.g., inadmissible evidence, or wrongful conviction). Therefore, with a focus on image forensics, this paper surveys techniques that can be used to enhance the robustness of machine learning-based binary manipulation detectors in various adversarial scenarios.


Introduction
As consumer technologies (e.g., image acquisition and editing tools) and artificial intelligence techniques advance, editing digital images and creating fake images are becoming easier and cheaper. Deliberate manipulation of digital images can be innocuous (e.g., to improve the quality and appearance of an image) or carried with malicious intent (e.g., to alter the semantic content of the image, or to establish an alibi). The diffusion of fake images has implications on judicial systems, global economy, financial health, and even homeland and national security. Not surprisingly, there have been interest from the digital forensics, and more specifically image forensics, community in recent years to detect deliberate manipulation of digital images. There have also been interest from the commercial market, as suggested in a recent study [1].
Image forensics, an emerging forensic discipline, seeks to determine the history of an image (e.g., its origin), the processing it underwent, etc, in order to determine the authenticity of the images [2]. In other words, key image forensic tasks include source classification (to reveal the source image produced from an image acquisition device, such as scanner and camera), source identification (to identify the acquisition device used to take the image), reverse engineering of processing operators (to identify the chain of processing operators, and processing includes deliberate manipulation), and authenticity verification (to determine whether the image has been manipulated).
In recent years, there have also been attempts to utilize machine learning (ML)-based techniques to both support different image forensics tasks and to defeat ML-based image forensics [3,4,5,6]. This reinforces the importance of developing techniques to protecting ML systems (also referred to as adversarial ML (Adv-ML)). Both ML and deep learning algorithms can be vulnerable to adversarial attacks that hinder their applications in security-sensitive domains, such as image forensics. For example, a deep neural network may report high confidence in a wrong prediction or can be circumvented by image perturbation techniques. Therefore, there have been attempts to design effective adversarial counter forensics (CF) techniques.
A number of literature surveys and reviews on the applications of ML-techniques in image forensics has been published in the literature [7,8,9,10] and in image forensics [11,12,13,14], although adversarial image forensics is generally not discussed. Amodei et al., [7], for example, reviewed the general security concerns in artificial intelligence, particularly reinforcement learning and supervised learning algorithms. A general review of security implications on the use of ML approaches and their countermeasures was presented in [8,10]. Akhtar et al. [9] focused on adversarial attacks on deep learning approaches in computer vision. However, there have been limited studies focusing on ML-security issues in (adversarial) image forensics, a gap we seek to address in this paper.
Specifically, in this paper we survey existing ML techniques for image forensics, including those that can be utilized in the adversarial setting (e.g., image manipulation), and CF. In the survey, we also reviewed the various approaches that can be used to enhance the security of binary manipulation detectors based on ML and defensive techniques during the testing stage. Figure 1 shows a graphical abstract concerning the application of machine learning techniques in adversarial image forensics. We will now explain the layout of the remaining of this paper. Section 2 introduces the reader to relevant background materials. In Section 3, we present the survey on ML-based image forensics, prior to reviewing adversarial image forensics in Section 4. In the latter section, we also revisit ML-based CF challenges and potential mitigation approaches (commonly referred to as anti-CF). Moreover, a classification of ML techniques in the presence of the adversary is presented. In Section 5, we conclude this paper.

Background
Here, we will revisit popular image forensics approaches such as Support-Vector-Machines (SVM) and Convolutional-Neural-Networks (CNNs), followed by ML-based image forensic techniques.

Image Forensic Analysis Approaches
The primary goal of image forensics is to identify remnants of any activities over an image file, which can be utilized by a forensic analyst in timeline or relational analysis and event reconstruction. Existing image forensic analysis techniques can be broadly categorized into those based on image acquisition, image coding, and image processing [2].
• Remnants of Acquisition-based Activities Every step of the image acquisition process leaves some traces within an image (e.g., traces for instance of the particular Color Filter Array (CFA) pattern and the type of filter used for color interpolation) [15]. Thus, the digital image produces evidence for interpolation filter and CFA pattern [16]. Furthermore, a camera sensor, by default, leaves a particular noise in each captured image, which is known as Photo-Response-Non-Uniformity (PRNU). In other words, two different image acquisition devices of the same make and model are likely to leave different patterns and the analysis of the traces will allow us to detect the image acquisition device used to acquire the image. Khanna et al. [16] detect footprints of digital tampering in lossless and lossy in compressed images. Table 1 summarizes existing forensic analysis methods designed to identify remnants of image acquisition activities.
• Remnants of Coding-based Activities Modern-day digital cameras generally utilize JPEG compression as a standard format for efficient storage and transmission, which partly explains the interest in the study of an image's compression history [17,18,19]. Since various imaging software applications generally consider different compression parameters and employ different quantization tables [17], the analysis of inconsistencies in quantization matrices can also be used for source and forgery detection. Ferrara et al. [15] proposed a method to distinguish aligned from not-aligned DJPEG compression by identifying the DCT coefficients of the first compression. Table 2 summarizes existing forensic analysis methods designed to identify remnants of image coding activities.
• Remnants of Editing-based Activities The use of image and video editing software will also result in the presence of digital footprints. For example,the application of a geometric transformation, such as a rotation or a resizing, requires interpolation of pixel values that leaves detectable traces in the image [2]. By employing a resizing and rotation geometric transformation, the software leaves detectable artifacts on the image due to value interpolation. Moreover, by altering the subpart of the image in terms of applying blurriness, contrast, saturation, etc, the software leaves artifacts in different areas of the image [20]. Median filtering in image processing, plus denoising and smoothing, can additionally be used to hide footprints of previous processing attempts [21]. Yuan et al. [21], for example, proposed a novel techniques for detecting median filter images through block-wise method by considering median filter between the blocks.

Image Forensics Tools
Now, we will review three widely used image analysis tools (see also Table 4).

Forensically
Forensically is a free tool in digital image forensics that offers features such as clone detection and metadata extraction [22]. The tool's magnifier or zoom factor helps analysts to find hidden features in an image by magnifying the pixels' size and color. Magnifier for detection uses the following three items based on histogram equalization, auto contrast, and auto contrast by channel. Clone detector in Forensically identifies manipulated areas of two or more image files. Similar areas are illustrated with blue and connected with red and overlapping areas with white color. Several options can be selected in this tool based on minimal similarity, minimal detail, minimal cluster size, block size, maximal image size, and show a quantized image. Error level analysis compares the pristine image with a compressed version to identify possible features lost during compression. Therefore, manipulated areas can stand out in various ways. This tool compares different features of image files such as color quality, error scale, magnifier enhancement, and opacity. Noise analysis is another part of Forensically, which is utilized as a median filter to identify noises in an image file that can be used for manipulation detection like airbrushing and wrapping. Level sweep in Forensically helps analysts detect copy-pasting areas by considering sweep through an image histogram. Furthermore, the copy-pasting areas are more visible due to the use of magnifiers. With the JPEG analysis tool, analyzers can extract metadata from JPEG files, such as quantization tables. Accuracy are achieved based on four different scanners: (S 1 ) Epson Perfection 4490 Photo, (S 2 ) HP ScanJet 6300c-1, (S 3 ) HP ScanJet 6300c-2, and (S 4 ) HP ScanJet 8250. 2D Reference Pattern, The accuracy of actual S 1 is predicted with 66.8% as S 1 and 33.2% as a S 2 . The accuracy of actual S 2 is predicted with 22.5% as S 1 and 77.5% as a S 2 . The accuracy of actual S 2 is predicted with 69.4% as S 2 and 30.6% as a S 4 . The accuracy of actual S 4 is predicted with 0.40% as S 2 and 99.6% as a S 4 .
1D Reference Pattern, The accuracy of actual S 1 is predicted with 63.7% as S 1 and 36.3% as a S 2 . The accuracy of actual S 2 is predicted with 21.6% as S 1 and 78.4% as a S 2 . The accuracy of actual S 2 is predicted with 85.1% as S 2 and 14.9% as a S 4 . The accuracy of actual S 4 is predicted with 0.00% as S 2 and 100.0% as a S 4 .   [20] In this scenario, the detection algorithm's performance increases with high parameter c. They achieved best performance when c = 112. The performance degrades when they consider c = 32. This scenario applies to contrast enhancement detection with γ = 1.1.
median filtering [21] No-JPEG and JPEG Images: ROC curves for MMF and f in uncompressed scenario demonstrates that features are more robust to detect median filtering. In JPEG scenario they consider 3 × 3 blocks and QF = [100, 90,80,70]. The performance degrades with QF > 80 but the performance improves when QF < 70. JPEGsnoop extract the hidden details of the compressed image and motion JPEG.
Detect only one specific compression method e.g. JPEG not TIFF.

Assembler
Recently on 5 January 2020, Google published Assembler which is a tool designed for journalists to detect forged images [23]. Assembler uses several existing methods to recognize common image manipulation detection (e.g., image enhancement, copy-move, and splicing). This tool also includes a detector to identify deepfakes, which is generated through StyleGAN. Assembler can help one identify which part of the image has been manipulated, determining copy-move and splicing in the presence of image brightness. While Assembler can help journalists spot manipulated images, it does not include other existing manipulation methods for detecting video and audio files. Moreover, this system needs to be real-time when used.

JPEGsnoop
JPEGsnoop [24] is a free software that can assess and extract hidden details of a compressed image, motion JPEG, and Photoshop files. JPEGsnoop is also capable of analyzing the image's source to obtain information from compressed images such as quality factor, and reporting on information such as chrominance and luminance quantization matrix, estimation of JPEG quality factors, Huffman tables, and histogram for RGB images.

ML-based Image Forensics
Unlike traditional image forensic tools, such as those discussed in Section 2.1, ML algorithms can be used to learn complex patterns from a set of hand-crafted features and facilitate classification.
• SVM-based image forensics SVM is widely used, partly due to its simplicity and accuracy outcomes in many classification tasks. In most of the earlier SVM-based forensic approaches, the features are hand-crafted, extracted from the image based on some heuristics, and very specific for the problem at hand (e.g., to detect double JPEG compression (DJPEG) and recompression). For example, Chen et al. [25] considered a set of features to improve DJPEG traces to discriminate between double and single JPEG images, which can then be used to train an SVM-based classifier. Nowroozi et al. [19] considered the traces left by DJPEG image in the mean, variance, and entropy, by training the SVM classifier using these collective statistical features. Milani et al. [26] presented a statistic derived from the DCT histograms based on first significant digits (FSD), in order to distinguish single from double JPEG compression.
Islam et al. [27] introduced a technique for splicing and copy-move detection that utilizes SVM classification of Local Binary Patterns (LBP) descriptors derived from the block DCT of chroma channels. SVMs have often been used for camera model identification, to fulfill multi-class classification based on high order features derived from the images or from the PRNU noise pattern generated by different cameras [28]. A summary of the different SVM-based image forensics techniques is shown in Table 5.
Other CF strategies could deceive this approach. [27] Splicing and copy-move detection. AUC decreases when window size is 4 × 4.
• CNN-based image forensics Deep learing (DL) methods such as Convolutional Neural Networks (CNNs) have also been used in steganalysis purposes, and generally the performance of these new CNN-based techniques exceeds those of classical model-based and standard ML-based techniques considerably.
The first task in image forensics utilizing CNNs for detecting median filtering is to apply feature representations with high accuracy compared with hand-crafted features methods [29]. The authors in [30] studied a binary and a multi-class CNN, which is effective for the different manipulation operations such as blurring, median filtering, and resizing. This approach was later extended in [31]. In each image patch, they found a strong difference between different camera models. However, these networks are partly shallow, consisting of only three or four convolutional layers. Besides, a pre-processing filtering step is employed to the first layer to force the network to look for the traces in the residual domain high-pass image, hence facilitating its job.
Niu et al. [32] computed a primary quantization matrix using DCT coefficients based on CNNs, which can work under a variety of situations. Also, they achieved good performance on small image patches. In summary, the method of considering completely self-learned features, without forcing the initial layers, has been shown to be useful, as long as sufficient training data is available. A summary of the different CNN-based image forensics techniques and ML-based approaches is presented in Tables 6 and 7, respectively.

Review of image forensics literature
Here, we will focus on addressing the following research questions:  [29] Median filtering detection, particularly in cut and paste manipulation.
Small image size detection with low quality factor. [30] Different manipulation detection such as blurring, median filtering, and resizing.
Problem with small window size, for example median 3 × 3.
[32] Compute primary quantization matrix in DJPEG. Tampering detection. Multi Purposes - [31,42] • What is the trend of ML applications in image forensics?
• How anti-CF techniques can be used to improve the security of ML against adversarial attacks?
• What methods are available to improve the security of ML engines used in image forensics?
We searched the literature using the following keywords: (image forensics) OR (image-forensics) (counter forensics) OR (counter-forensics) OR (counterforensics) OR (anti-counter-forensics) OR (anti-counter forensics) OR (anti counter forensics) on Google Scholar, IEEE Xplore Digital Library, Springer, and ScienceDirect. The searches were limited to the paper title, keywords, and for conference, journal, and magazine articles.
We only included English-language studies that report on empirical and theoretical findings in image forensics using machine learning. We also considered patents. Table 8 describes our inclusion criteria.

Content Expectation
Empirical and theoretical papers related to image forensics Papers focusing on forge image detection.
Empirical papers related to image security Papers focusing on CF and anti-CF .
Machine learning on image security Papers focusing on different attacks and defensive methodologies .  As shown in Figure 5, "adversary", "robust" and "security" are three popular keywords. This suggests a growing interest in studying the security of image files in the presence of an adversary.

Knowledge Synthesization
The primary keyword in Figure 5 shows that there are important papers that are related to robustness, security, and classifier. Based on this knowledge, the selected subjects only focused on the solutions to today's difficulties, and the few practical tools. Some practical security explications in ML, particularly for image forensics, only introduced a method for answering the specific problems in this area. Therefore, based on our knowledge of protecting the detectors against adversarial attacks requires a specific architecture that works against different attacks is infancy. Moreover, security and untrustworthy in ML techniques for forensic image detection become important.  The studies showed that the security of ML techniques in the presence of an adversary plays an important role, and the development of solutions capable of improving security becomes necessary. In the following, to make this survey more focused, we considered three research questions.

RQ1: what are the latest machine learning applications in image forensics?
Image forensics' task has been deal with numerous algorithms based on statistical analysis and pattern recognition-lately, computing capabilities' improvement further attention in ML methods and particularly in deep learning, and have demonstrated their effectiveness in many image forensics competitions [11,12]. Techniques based on ML/DL-based image forensics have been great attention in recent years. Methods based on DL have consistently achieved remarkable results on a series of tasks in image forensics such as Face GAN detection [43], Deepfake detection [5,14], Coding-based detection [25], and so on. Among them, CNN's are effective when dealing with various image tasks and utilized as a base for many image forensics techniques. One example related to median filtering detection is considering CNNs [29] and utilizing different detection tasks such as blurring, noise addition, resizing, rotation, etc. Besides DL, in many complex tasks, researchers considered ML methods to address different forensics tasks, e.g., SVM. SVM-based in image forensics was recently utilized for detecting different forensics tasks, such as align and not-align DJPEG [19], global and local contrast enhancement [34], and so on. Therefore, RQ1 plays a pivotal role in knowing new methodologies in adversarial image forensics, e.g., such as counter-forensics and anti-counter forensics.

RQ2: how anti-CF techniques can improve the security of ML against adversarial attacks?
Anti-forensics or counter-forensics are methods to make forensics algorithms fail by altering the image content or changing some forensic algorithm. The main aim of the forensic community is to develop more robust forensic schemes. Recent studies in DL led to the development of generative adversarial networks (GANs) [4] that confirmed to be more effective for misleading many current image forensics approaches, referred to as adversarial examples. Adversarial examples employ small perturbations on the image to indue a system to make a wrong decision. The perturbation can be obtained by estimating the input image's loss function, such as the FGSM method [4], Deepfool [44], JSMA [45], and so forth. Guera et al., [46] considered FGSM and JSMA adversarial attacks to mislead CNN camera model identification. Tondi [47] also considered a gradient attack on the pixel domain to create adversarial examples in the integer domain against CNN-based detection. According to the study, Most of the DL methods are fragile and vulnerable against adversarial attacks. The researchers report that even if the robustness of DL based method can be improved by retraining the classifier with adversarial examples, the resulting networks are still vulnerable against most powerful attacks. Anti-counter forensics or counter-counter forensics have been extended as a defense methodology versus counter-forensics methods by developing image forensics' security in counter-forensics attacks. While with the advent of adversarial attacks, various techniques have newly been developed in image forensics to defend versus adversarial attacks. Barni et al., [33] proposed a secure method to react against different post-processing by retraining unaware classifiers with the most powerful attacks (MPAs). Therefore, by understanding most CF attacks, the analyst can be developed a secure system to react against the most powerful CFs.

RQ3: what methods are available to improve the security of ML engines used in image forensics?
One of the hot topics in image forensics is developing methods for improving ML against different adversarial attacks. Most of the methods are tailored versus particular CF attacks. In this survey, we study the solutions suggested so far to counter CF attacks. Moreover, we distinguish the methods based on adversaryaware system and generally more secure detectors. In the first category, the analyst thought to be aware of the CF method the system is subject to and tries to develop a new algorithm to reveal the attack by looking at specific traces left by the tool. For instance, Barni et al. [33] proposed a detector based on a support-vector-machine (SVM) that fed with a large number of features to identify the traces left by D-JPEG in the presence of attacks. With the advent of DL architectures, adversary-aware training has been widely used to improve DL models' robustness to adversarial examples [4]. Although these approaches effectively improve the security of ML in image forensics, it still works only against specific attacks. In the second category, generally more secure detectors, most anti-CF methods are developed without considering developing a technique that analysts react appropriately against various kinds of attacks. Barni et al. [48] proposed a detector against PK attacks through multiple-classifier architecture. The architecture is also known as a 1.5C classifier, consisting of one 2C classifier, two 1C classifiers, and a final 1C classifier. They access the performance of the 1.5C against three different manipulation tasks resizing, median filtering, and contrast enhancement. In this survey, we study different methodologies proposed so far based on the above categories, and we address the pros and cons of methodologies.

Adversarial Image Forensics
With the advancement of forensics methods for retrieving information and tampering detection, in recent years, CF methods have been developed to prevent a correct detection. CF tools are usually effective due to the defects of forensic tools (most CF attacks take advantage of the weakness of the traces), which most of the time are not though to work under adversarial conditions. In real-world situations, the behavior of an adversary intending to stop the analysis cannot be ignored. Therefore, forensics analysis needs to up its game by improving its detection methods capable of working in an adversarial condition.
The task of improving forensics methods in an adversarial setting is even more challenging when MLbased, and in particular DL-based, forensic tools are adopted for the analysis. This challenge can be attributed to the inherent vulnerability of these tools, and the performance decreased on the different conditions during the testing phase, concerning those used for the training. This calls for the improvement of more reliable ML-based forensic tools, which is called a new class of tools that can efficiently counter-CF attacks (referred to as anti-CF) while keeping the advantages of modern ML methods [49]. The security of ML systems in the adversarial conditions is a common problem in comparison with many other security-sensitive applications. Hence, the use of comparable solutions should be considered to secure image forensic methods. Some useful symbols used throughout this section are listed in Table 9.

Counter-Forensics and Anti-Counter-Forensics
CF or Anti-Forensics, refers to all the solutions proposed so far to bypass the forensic analysis. Early proposed models of CF methods were fairly simple, including applications of basic processing operators [50,51,52]. By adding noise dithering, the gaps in the histograms of DCT coefficients are decreased, and the blocking artifacts are concealed by employing a smoothing operation, to hide traces of JPEG compression [52]. For instance, the most harmful attacks in image forensics related to the geometrical and CF attacks to eliminate the traces from previously JPEG images [53] (see Figure 6). In this case, the detector, in an unaware case, totally fails to detect.
To conceal traces of contrast enhancement operation, the authors in [51] proposed a method that removes picks and gaps in the pixel histograms through dithering. To hide artifacts of resampling, image high frequencies are perturbed with noise while being resampled [50]. Such techniques can be easily circumvented by dedicated methods.
The effective CF techniques can be devised by an attacker whenever he knows some information concerning the forensic algorithm. Such methods are assigned to as targeted attacks and tailored to a particular algorithm. CF methods are not perfect and leave traces on their own, that can be exploited by an informed  Figure 6: Scheme of the CF attack [53] analyst. The techniques developed by the analyst to defend against CF attacks are referred to as anti-CF techniques.

Counter-Forensic Attacks
We address useful terminologies to categorize the attacks in ML-based environments on influence, specificity, and security violations, as depicted in Figure 7.
• Influence: -Causative: The attacker feeds malicious input to the training data. In this case, by feeding adversary samples during training, the learning parameters of models will be changed. Therefore, it can influence the performance of the classifiers. -Explorative: The attacker feeds malicious input to the test data and never attempts to modify trained classifiers. The attacker tries to make a misclassification error through adversarial examples or seeks to obtain information from models.
• Specificity: -Targeted: When the attacker attempts to decrease the performance of the classifier by the deception of a specific algorithm or specific group of samples.
-Indiscriminate: When the attack has a more flexible goal, or when the attack is targeted to a class of algorithms rather than a specific algorithm.
• Security violation: -Integrity: When the attack aims at letting malevolent samples be classified as normal, therefore attacker tries to increase false negatives error when classifying adversarial examples on the classifiers.
-Availability: When the attack aims at causing a classification error of any type, i.e., both a false negative and a false positive error, thus causing a Denial of Service (DoS) concerning pristine samples.
Different kinds of attacks to the general ML system can be employed based on the above taxonomy, as done in [48].
In the following, we study the general adversarial model utilized for CF attacks. Then, we address some examples of several attack models based on perfect and limited knowledge attacks. Moreover, we review different adversarial attacks against DL forensic techniques.

Counter-forensic attack model
By following the terminologies in [48], an adversarial model is represented by specifying the assumptions about an attacker's goal, knowledge, and capability to corrupt the data or system.

• Attacker's goal
In this category, the attacker defines the kind of security violation. The attacker aims to apply integrity attacks that induce false-negative error of the classifiers or employ an availability attack to induce errors of classifiers such as false-negative and false-positive error. We can classify CF into violation attacks or evasion attacks whenever the attacker modifies the manipulated images so that they are misclassified by the detector, either assumed as pristine ones or become misclassification errors. In doing so, the attacker normally wants to introduce a small distortion into the image to cross the boundary of a decision to minimize visual distortion while maximizing the loss function.
• Attacker's knowledge The knowledge of the attacker can be classified based on Perfect Knowledge (PK) or Limited Knowledge (LK), by examining the attacker knows features, training data, classifier architecture, learning parameters, and decision functions [54]. In the PK scenario, an attacker has the full knowledge regarding the forensic algorithm. This is the most convenient case for the attacker. Conversely, in the LK scenario, an attacker has only a few bits of information concerning the forensic algorithm, e.g., may not be aware of the exact algorithm or some of the parameters of the algorithm l i ∈ L. In ML-based techniques, the attacker knows only somewhat of information about the training data D. Based on the available knowledge, the forensic algorithm specifies the specificity of attack based on targeted or indiscriminate. Recently, universal approaches are developed in such a way to be useful versus a whole class of forensic classifiers [55]. Barni et. al, [55] proposed a CF method that performs multiple compression undetectable based on investigating the histogram DCT coefficients. This approach removes double and multiple compression artifacts, while the visual quality of the image keeps high.
• Attacker's capability In this case, the attacker can be controlling training or testing data, which referred to as influence of the attack. We can interpret the impact of the attacker's capability whenever applied based on causative or explorative attacks. In the exploratory attack, an adversary may modify test data, but can not alter training data. On the other hand, in a causative attack scenario, the training process can be modified by an attacker, which referred to poisoning attacks. Moreover, most of the CF attacks in nature are exploratory [54].
The above explanation of the attack lets the attacker employ the threat model. Thus helps the analyst to design proper methods that can work in an adversarial situation.

Perfect knowledge attacks
The attacker in the PK scenario can employ his/her attack by relying on a forensic algorithm, denoted as φ, and then lunch the attack to the target [56] (see Figure 8). To keep a visual quality of the attack image, the attacker must induce an integrity violation, which causes to false-negative decision error. Particularly, the attacker requires to solve an optimization problem by looking at the image which is closest to the image under the attack. Generally, an attacker needs to look at the image, which is resembling more to the image under the attack by solving an optimization problem. Pasquini et al., [57] employed a CF method to counter DJPEG compression detection by relying on First Significant Digits (FSD). Particularly, Comesana et al., [58] shown the optimal attack against Benford's law-based detectors. Fontani et al., [59] considered a method for hiding the median filtering artifacts and, in another case, counter SIFT-based copy-move detection, which is close to CF attacks.
The optimum attack can be a gradient-based attack applied based on gradient-descent solutions when the detector is more complicated. To provide an example, Chen et al., [60] proposed a gradient-based attack employed on the pixel domain to counter SVM manipulation detectors on residual features. Similarly in [47], a gradient analysis is introduced to counter forgery detection. When small perturbations applied to an image, the existing techniques tend to be canceled by pixels rounding, thus causing an attack useless. Therefore, the attack in [47] generates adversarial images with small perturbations to cause the classifier for the wrong decision.
The main challenge with many PK scenarios is that most of the CF attacks are utilized in the feature domain. In this case, controlling the distortion will become a challenging task since the relationship among the pixel and feature domain is usually non-invertible [57] (see Figure 9). Generally speaking, a suboptimum

Limited knowledge attacks
We consider the taxonomies were introduced in [49], already employed to classify CF attacks based on universal attacks, attacks on a surrogate detector, and laundering-type attacks.

• Attacks on surrogate detector
In this case, the knowledge of the attacker for the algorithm φ is limited, and maybe he/she is aware of the X, the feature space, but the attacker does not know all the parameters of the algorithm L and also training set D in the case of ML-based techniques. The concept is shown in Figure 10. In Figure 10: Scheme of the attack on the surrogate model this case, the attacker creates a surrogate detector (φ) and tries to guess the parameters he/she does not know based on whatever information that is available to him/her. Afterward, the attacker applies the CF by performing the attack versus (φ), expecting the attack will work against the real detector, which is commonly referred to as attack transferability [61,62]. Recently, researchers prove that most of the MLs are fragile and vulnerable against adversarial attacks. Papernot et al., [61] confirm that the attack on the source network entirely transferable to target networks. Whereas, Barni et al., [62] argues that adversarial attacks are not transferable from source to target in image forensics application.
For instance, let l 1 , and l 2 be the unknown parameters, then the attacker tries to guess the parameters l 1 , and l 2 though his/her parameters arel 1 andl 2 andφ whereL = {l 1 ,l 2 , l 3 , l 4 ...}. The effectiveness of the method afterward is evaluated against the φ detector [49].
An example of an attack on the surrogate model is fingerprint-copy attacks for PRNU camera identification. For the attacker, K a real camera fingerprint is unknown, and then he/she tries to guess the parameters K andK based on available images. Many attacks on ML-based detectors fall into this scenario. In fact, the attacker has limited access to the D, but he/she is aware of the architecture [49]. Then, the attacker buildsD as another dataset with the same distribution of D and then employs it inside the pristine one. In other words, the attack duplicates information from the detector φ [60,54]. Chen et al., [60] applied a pixel domain attack against ML forensic image detectors by considering a large dimensionality feature vector. They considered uncompressed images for this scenario because compressed images tend to erase the traces introduced in the pixel domain. Thus, attacks on compressed images need to go more inside the decision region, leading to high distortion on the image. More specifically, Biggio et al., [54] proposed a technique for evading classifiers via discriminant functions, which is effective in PK and LK scenarios.
• Universal attacks In this category, the attacker does not know exact statistics performed by the analyst but is aware of the X features; therefore, the attacker employs an effective attack against φ inside the class of φ (φ equal to φ(L , X ), ∀L ). Figure 11 shows the universal attacks' procedure. Some of the examples Figure 11: Scheme of the universal attacks are applied based on first-order statistics of the image versus the class of detectors to fool contrast enhancement [63] . For instance, Alfaro et al., [63], considered a method based on the first-order statistics in the DCT domain to deceive DJPEG detection.
• Laundering-type attacks The attacker only knows quite general and limited knowledge regarding the algorithm (see Figure 12); then the attacker employs basic processing operation to erase the traces left by CF, such as considering filtering, resampling, and other processing. Moreover, the target of the attack does not limit only to a specific class or a detector. Early laundering-type attacks have been developed against resampling detection [50], S-JPEG compression [52], and contrast adjustment [51], to name a few as pointed out in Section 4.1. In the literature, such attacks are usually related to targeted attacks, then they could be included in the PK scenario.
The reason why these approaches such as contrast adjustment [51], double JPEG compression [52], are not included in PK attacks by [49] is the knowledge of these algorithms which is only marginally employed with basic processing which turns out to be enough for the attack purpose without any optimization. In most cases, the algorithm φ only utilized to prove the attack's effectiveness and not to guide the attack.
Generally speaking, the implementation of such CF attacks is much easier than most PK attacks because the image distortion can be easily controlled by the attacker.

Attacks on deep learning-based image forensics
CNN architectures are capable of learning complex forensic features directly from input data or directly from an image. In contrast, an intelligent attacker can utilize this property and employ the most powerful CF attacks. In recent years, researchers in the ML area have found that the DL methods could be vulnerable and fragile against adversarial attacks, including the forensics area. An attacker can create modified images and force misclassification errors due to the difference in the space of the inputs and images used to train the CNN, referred to as adversarial examples [64].
Adversarial examples utilize the above property by applying small perturbations on the image to induce a system to make an incorrect decision. In this case, an adversarial image is generated, which is visually indistinguishable from the pristine one and misclassified by CNN. Adversarial perturbation can be obtained by estimating the gradient of the loss function of the input image, such as the Fast-Gradient-Sign-Method [4], and DeepFool [44]. Also, other attacks can be achieved by using iterative methods, including the Jacobian-Based Saliency Map Attack [45], or by box constrained L-BFGS [64]. Adversarial examples to DL models are applied at testing time, which belongs to the category of explorative attacks and PK scenario. Adversarial examples in the PK scenario is often referred to as the white-box scenario in DL literature. Whereas, the LK scenario is related to as a gray-box scenario [65]. In a realistic and challenging scenario, black-box attacks are also considered in DL literature when the internal details (e.g., parameters) are not accessible to the attacker. For this reason, the attacker may employ several queries to gain internal details.
Recently, CF methods have been developed against CNNs in image forensics. Guera et al., [46] considered the CF attack to deceive a camera model identification by considering FGSM and JSMA to mislead CNNbased camera model identification.
Rounding process to integers is sometimes enough to wash out the perturbation and make an adversarial example ineffective. Tondi, [47], applied a gradient attack on the pixel domain to create adversarial examples in the integer domain against a CNN-based detection.
The transferability of adversarial examples is the main concern in the security of ML-based image forensics, particularly in CNNs. Barni et al., [62] proved the adversarial examples are not transferable against CNN-based detectors. Whereas, Li et al., [66] proposed a new method to improve the strength of the attack to assess the transferability of adversarial examples. As a result, they created a strong attack by enhancing the confidence of the misclassification.
In the following, we review recent adversarial attacks applied to ML, particularly on DL. The study mainly deals with methods that try to deceive the deep neural networks.

• L-BFGS adversarial attack
The authors in [64] proposed adversarial examples against the neural network in 2014, and they generated adversarial examples using box-constrained LBFGS. This method looks for an adversarial image (X adv ) with regard to input image X under the L 2 distance, yet the label is different from the classifier. In X, we consider y as a ground truth class, then X adv is s such that l(X adv ) = y, that is, l(X adv ) = 1 − y, where l() indicates the class label (then, l(X adv ) = 0 if y = 1). The constrained minimization problem is a challenging task to solve but can be formalized as follows: An approximately optimum solution can be used to solve the relaxed problem [64]: which J θ (X, y) referred to as loss function such as neural network cross-entropy with parameter θ and scaler c. Moreover, the gradient-descent algorithm is utilized to (2) to solve the problem. The gradient-descent algorithm is applied to solve (2). To find the constant c > 0 line search is considered at a minimum distance to yield an adversarial example. Parameter c is solved with different values through the bisection search of other techniques for one-dimensional optimization. Based on the above formalization, an adversarial attack can compute perturbations and then apply to X to fool a neural network. In this case, the human visual system can not distinguish between X adv and a clean image of X. An example of this scenario is depicted in Figure 13. • Fast Gradient Sign Method Goodfellow, Shlens and Szegedy [4] proposed a fast, suboptimum adversarial attack, called FGSM. Given an image X, the FGSM method is formalized bŷ where a parameter ε in (3) referred to as the strength of the attack, which has a small parameter that leads to undetectable. Intuitively speaking, this method determines in which direction the pixels should be modified by considering the gradient of the loss function. Figure 14 shows an adversarial example that is generated by a Fast Gradient Sign Method attack [4]. In comparison with the previous attack in Section 4.2.4, L-BFGS, the FGSM adversarial attack is devised to be optimum and fast and has more efficient computation, yet, it does not create a close-to-minimal adversarial perturbation. Another difference with respect to L-BFGS, which is optimized for the L 2 distance metric, the FGSM method is optimized for the L ∞ distance metric.
Sometimes, an attack based on (3) fails to get the adversarial images. So, another less suboptimum attack can be achieved by considering the iterative version of the attack. In each iteration i + 1, the adversarial perturbation is obtained as specified in (3), and the image is updated as Moreover, in the direction of the gradient sign, the attack is employed iteratively i with a small ε. • Jacobian-based Saliency Map Attack JSMA adversarial attack is a greedy iterative method relying on the forward propagation, proposed by [45]. This attack is optimized under the L 0 distance metric. A saliency map is computed in each iteration for the pixels that have a high contribution to the classification, and then based on the saliency map, sensitive pixels are modified by the parameter of θ < 1. Hence the modification of the pixels being, The iterative process ends when either 1) adversarial examples are misclassified, 2) the attacker succeeds by getting an adversarial X adv such that ll(X adv ) = y, or 3) whenever an adversarial image cannot be found for a given maximum L 0 distortion (sample of saliency map is illustrated in Figure  15). • Projected Gradient Descent PGD adversarial attack determines the perturbation under constraints L ∞ that maximizes the loss function [68].
In each iteration i + 1, X first is updated according to some rule X i+1 . Then, the image is projected onto the space having constrained L ∞ distortion with the maximum distortion set to value α, that is, In (6) Π is the projection operator. PGD is an iterative extension version of the FGSM and is similar to the I-FGSM.
Based on [69], PGD adversarial attack is implemented as follows. Multiple times a gradient sign attack is employed with a small step size ε (ε < α); for pixel (R, C), where X i+1 uses the equation of (4).
• Attack on One-Pixel Another scenario to deceive a classifier is based on the one-pixel attack. Su et al., [70] demonstrated can successful deceive three different models by modifying one pixel per image through the concept of Differential Evolution [71]. In their work, they received 70.97% accuracy of the tested images with the average confidence of the network of 97.47% on the wrong labels. Figure 16 shows an example of the adversarial attacks on one-pixel. For an image X, first, the vectors R 5 are generated, including Figure 16: An example of one-pixel adversarial attacks [70] RGB values for an optional candidate pixel and xy-coordinates, then details are modified randomly to generate children vectors and including a child encounters and parent iteratively.
In comparison with previous attacks, differential evolution creates adversarial examples without any access to gradients, network parameters, and values.

• Carlini and Wagner Attacks
Carlini et al., [72] explained that defensive distillation does not improve the robustness. To this end, they proposed three new attack algorithms and demonstrated that the defensive distillation completely fails against these attacks. They argue that their attacks are more effective concerning the three distance matrices L 0 , L 2 , and L ∞ norms, as they called it the C&W attack. Moreover, they proposed high-confidence adversarial samples in a transferability scenario that can defeat defensive distillation. The C&W attacks can compute perturbations for a black-box scenario. Figure 17 shows examples of adversarial examples produced by C&W adversarial attacks.

• DeepFool
Moosavi-Dezfooli et al., [44] initializes the image X that is restricted by the decision boundaries of a Figure 17: Example of C&W adversarial attacks [72] classifier. The region of the decision boundaries determines the class label of the image. In each iteration, a small vector applied to an image that is estimated by the boundary of the polyhedron. Then, in each iteration, the perturbations are applied to an image to measure the final perturbation based on the original decision classifier boundaries. The underlying algorithm is optimized for the L 2 and L ∞ norms.
A summary of the main adversarial attacks recently applied to DL-based image forensics is provided in Table 10.
Perfect [58] Optimal attack against histogram where detection region is non-convex.
Considered only the MSE.
Perfect [59] Conceal traces of median filtering. Degrade the performance against JPEG compression.
Perfect [60] Attack against SVM of global image manipulation based on gradient descent.
With JPEG compression attacker needs to apply more distortion.
Limited [61] Attacks on surrogate detector: adversarial sample transfer across MLs.
Transferability can be improved by increasing the attack strength.
Limited [62] Attacks on surrogate detector: Apply the scenario [61] in image forensics.
Integer domain.
Can not applicable to other histogram based detectors.
Proposed methods are not detectable with other existing forensic methods.
Limited [52] Laundering attacks: remove JPEG artifacts. Designed for specific scenario.

Generative Adversarial Networks (GAN)
With adversarial examples, CF scholars in the image forensics community have started studying Generative Adversarial Networks (GANs) [73]. GANs are developed to generate generative models by imitating the distribution of training data. This is employed iteratively to make a min-max game between two-players, that is to say, discriminator train iteratively to distinguish between real and generated examples to deceive the discriminator. Newly, GANs have been employed as a CF attacks. For instance, Kim et al., [74] present median filtering CF attacks by considering GAN network, which can effectively erase the traces of median filter images. Bonettini et al. [75] considered Benford's law to discriminate generated images from pristine ones. Benford's law shows the distribution of the most important digit for DCT coefficients.

Summary of counter-forensics attacks
We summarize the different CF methods applied to an image to bypass the forensic analysis. Particularly, the problem of CF is discussed, and the related prior art is shown. As shown in Table 11, an attacker with a perfect knowledge can make the attack by relying on the knowledge of the forensic algorithm. One the other hand, in a limited knowledge scenario, the attacker knows only some parts about the forensic algorithm, which means that it does not know the specific algorithm or maybe parameters.

Anti-Counter Forensics (Anti-CF)
Anti-CF methods have been extended to react to CF attacks and restore the validity of the forensic analysis. Most of the anti-CF methods are tailored against specific CF methods. Anti-CF methods can be classified into two categories: adversary-aware detectors which is the most common case in image forensics, and generally more secure detectors [49] when the attacker designs a secure system which is intrinsically difficult to attack.

Adversary-aware detectors
This is the most common approach in Adversarial Image Forensics, particularly in ML approaches. The analyst assumed to be aware of the CF attack; therefore, design a new method to expose the attack by looking at specific CF footprints. Then the new algorithm φ A is devised within conjunction with the original algorithm φ.
Barni et al., [33] proposed a detector based on adversary-aware training to detect dangerousness CF attacks and other processing in the presence of DJPEG compression. They considered the Most Powerful Attacks (MPAs), which degrade the classifier performance in an unaware case. Then, a detector retrained to identify images subject to MPAs should recognize milder processing and double compressed images. The example of this scenario is depicted in Figure 18. In other suited cases to ML-based is when the new algorithm Figure 18: Refined the decision margin with attacked samples (red dots) and close more to pristine samples (blue dots) [33].
is obtained by employing an adversary-aware version of φ A and is utilized in place of φ, indicating that the algorithm can be retrained by attack samples [33,35,34]. In this way, the detector achieves the refined version of φ A = φ(L, X ; D ∪ D A ) by retraining the set of attacked examples D A . In general, this method works properly whenever the classifier can distinguish enough between pristine, manipulated, and adversarial examples. To provide an example, the authors in [34] proposed a system capable of detecting contrast enhancement in the presence of JPEG compression. In this system, different SVM classifiers (unaware case), trained with the compressed images. The second approach is to estimate QFs by utilizing the idempotency of JPEG images whenever QF cannot extract from the JPEG header. Hence, different classifiers trained with different QFs separately, and based on the idempotency approach, the nearest QFs selected as a detector. Then, Barni et al., [35] improved the approach in [34] based on CNN detector for generic contrast adjustment against JPEG compression. In this system, robustness achieved by retraining unaware cases with JPEG examples with different QFs. Experimental results prove this system also can work under unseen tonal adjustments.
Furthermore, recently, forensic analysts extensively start to study how they can improve the robustness of DL based on adversary-aware training against adversarial attacks [4]. Want et al., [76] proposed a technique to prevent adversarial dangerousness attacks on DL based on multi-source and multi-cost schemes for improving defense performance, called Adversarially Trained Model Switching (AdvMS). Based on the AdvMS scheme, the first component multi-source alleviates the performance problem, and the second component multi-cost enhances the robustness.
The advent of Generative Adversarial Networks (GANs) develops many challenges in image forensics. GANs have been extensively employed to generate fake images. Identification of GANs-generated images is a significant forensics challenge, particularly for Image-to-Image translation and DeepFakes. Nataraj et al., [77] proposed a new method for identifying GAN images using a combination of different co-occurrence matrices. They consider matrices on three color channels RGB and train a model using CNN. The challenging task in this scenario is that the proposed methodology intrinsically is not robust against JPEG compression leading to the network being re-trained with different JPEG quality factors. In this work, they obtained 93.78%, 91.61%, and 87.31% for quality factors of 95, 85, and 75, respectively.

Generally more secure detectors
Most of the current anti-CF methods do not take into account the possibility of predicting the movement of the analyst and determining the probable course of action for each attack. When the attacker predicts that CF footprints are left by themselves, they devise more robust CF methods that leave less evidence, leading to a loop between CF and anti-CF techniques.
A possible solution concerning this problem is to devise a system that resists against PK attack and inherently difficult to attack. In compered with adversary-aware training, the analyst makes a system to counter several CF attacks, for instance, one possible approach is to consider higher-order statistics. In this case, X is the set of features from the original algorithm, andX is a larger feature space that estimates by the algorithm, whereX ⊃ X . This method was implemented by Rosa et al., [78] by applying secondorder statistics for contrast enhancement detection. The security of the proposed system in [78] evaluated the universal CF scheme [55]. They achieved three interesting facts: 1) This system is slightly better than the first-order detector; 2) CF attack methods recently applied against first-order statistics detectors are not functional versus second-order statistics; 3) Evaluate traces that left by an attack; it is easy in second-order statistics. Furthermore, a similar approach was proposed for countering S-JPEG, DJPEG, and local tampering anti-forensics in [79,25]. Considering second-order statistics reveal CF attacks, helping the analyst to perform a more correct analysis. Another strategy to design a generally more secure system based on fusing the outputs of different forensic algorithms by looking at traces [80]. Researchers borrowed this scenario based on Dempster-Shafer and investigated how Counter-Anti-Forensics (CAF) tools can be embedded in this system for classifying CAF traces from image forensic traces.
General speaking approaches referring to the category of generally more secure detector look for solutions in the presence of the worst-case, for a given class of attacks. An example of such a technique can be found in [81] for detecting DJPEG compression by re-training on D ∪ D A * , where A * is considered as an optimum attack. In the following, we present an overview of recent works developed for improving the security of detectors.

• Data Randomization
Randomization strategy is another method in this category that can be employed to enhance the robustness for forensic detectors on DL for general models and standard-based forensics [82,83]. Zhang et al., [82] they optimized the feature sets, which intrinsically become secure against a PK attack by considering feature selection method on adversarial samples that increase the security versus attacks at test time. Therefore, feature randomization plays an important role in the security-related application when a small set of features are considered to overcome the complexity or even enhance the performance of classification to tackle adversarial attacks. Barni et al., [83] considered random feature selection strategy that can improve the security and robustness of forensic detectors for the general model and standard ML-based to mitigate the adversarial dangerousness attacks. The experiments prove that feature randomization strategy reducing the transferability of attacks, and increase the security of detection even in the presence of mismatch architectures. Moreover, such techniques also have been confirmed to be effective against PK attacks [84].

• Defense Layer
In another strategy adding a new defense layer in a network helps to counter adversarial attacks in the back-box setting. In this approach, the defense layer parameters assist in gaining protection versus adversarial attacks (see Figure 19) [85].
• One Class Classifier One class classifier (1C) has been recently used by the forensic community to improve the security and robustness of image forensics, particularly in security applications. The main idea behind using autoencoders in image forensics is that they play as a role of 1C classifiers, which forgery data can be defined as an anomaly. More in general, in many different applications, 1C modeling is famous for anomaly detection, when a statistical characterization under abnormal situations is not feasible. 1C combinations were devised for improving the security versus evasion attacks for adversarial anomaly detection. Moreover, 1C classification is an alternative approach for conventional multiclass algorithms Figure 19: Defense layer architecture [85] that classify the examples based on several pre-defined categories, which referred to as open set conditions. In particular, in several forensic tasks and security-oriented tools, the open set problem has been investigated so far in [86]. Wang et al., [86] considered SVMs multi-class and 1C to recognize different camera models. This technique is more robust against DJPEG images. In image security, 1C classifiers have also been developed in association with GAN to design detectors when malicious examples in training exist. Yarlagadda et al., [87] considered this methodology for satellite images to evaluate their authenticity. In this architecture, GAN learns pristine satellite features, and 1C (SVM) is trained with those features. As illustrated in Figure 20, the rational idea behind discriminator is to distinguish patches from a generator and pristine satellite images accurately. In contrast, the generator intends to deceive the discriminator by generating data near a pristine one. Figure 20: Architecture satellite Image foorgery detection [87] • Ensemble Method Multiple architecture combination (1.5C) is another strategy to improve the security and robustness against PK attacks in image manipulation detection, which decreases damage caused by an attacker [48]. The 1.5C architecture is achieved by a combination of 2C and 1Cs classifiers, runs simultaneously, and outputs of each classifier feed to the final 1C classifier. The logical idea behind scenario 1.5C is as follows. 2C classifiers always generalize entirely to the samples that were not represented correctly in training but achieved high accuracy and performance, especially in the absence of attacks. Therefore, based on this problem, an intelligent attacker can use this property and applied the attack and fall into 'unseen' regions, which is illustrated in Figure 21 [81]. In Figure 21, the attacker aims to take manipulation samples (red triangles) and transfer to a pristine unseen region (blue dots). By enclosing only one class, the system will become robust and secure against adversarial attacks. This behavior is depicted in Figure 22. We see that moving samples from manipulated to the pristine region need a more distortion because of the adjacency of pristine samples to the decision region. To achieve a Figure 21: Scheme of 2C when the attacker transferred 'red triangles' to unseen 'blue dots' region [48] Figure 22: Scheme of 1C by defining a closed region [48] system that is intrinsically difficult to attack, the classifiers are combined, which has similar accuracy with respect to 2C and high accuracy with respect to 1C. Based on Figure 23, the acceptance region is well-shaped compared to 1C with similar accuracy with respect to 2C [48]. Recent advancements in technology have created realistic images via GAN as to be imperceptible Figure 23: Scheme of 1.5C by defining a well-shaped decision region [48] from the pristine ones. Rana et. al. [88] proposed deep ensemble methods for detecting Deepfake videos by combining various DL classification models.
• GAN Defense Lee et al., [89] directly train a network besides the generator and tries to create the perturbation for that network. Thus, the classifier can correctly distinguish between pristine (clean) and manipulated one (perturb image). The proposed approach is robust against adversarial attacks such as FGSM [4]. Defense-GAN is another methodology defending classifiers versus adversarial attacks by training the model with the distribution of unperturbed images [90]. The system tries to obtain a close output, which does not include the adversarial modifications. Therefore, this system can be employed as a defense technique against any attacks in the LK scenario. While modern GAN can create fake images with high quality, re-construction of relations among color bands is expectedly difficult. Barni et al., [43] proposed a methodology for detecting GAN generated face images via co-occurrences analysis. Table 13 summarizes the various attacking and defensive ML-based techniques, and Table 14 presents experimental findings of the approaches described in Table 13. Adv-aware detectors [33] Detect dangerousness attacks and other processings in the presence of DJPEG compression.
Adv-aware detectors [76] Prevent adversarial dangerousness attacks on DL by employing multi-source and multi-cost schemes.
Reduce the performance with large .
Adv-aware detectors [77] Identifying fake GAN images using combination of different co-occurrence matrices.
Performance degrade on high quality images such as StyleGAN1 or 2.
Generally more secure detectors [78] Contrast enhancement detection by employing second-order statistics. Performance degrade against JPEG compression.
Generally more secure detectors [81] Detecting DJPEG compression. Works only with SVM.
Generally more secure detectors [83] Data randomization: improve the security in different forensic scenarios.
Adversarial examples can be transferable by increasing strength of attacks.
Generally more secure detectors [85] Defense layer: Counter adversarial attacks.
It cannot work in a white-box scenario.
Generally more secure detectors [48] Ensemble Method: Protect detectors against PK attacks.
Not to much robust against: 1) Noise addition 2) JPEG compression.
Generally more secure detectors [89] GAN Defence: Robust against adversarial examples using a GAN.
Problem with deep network.

Summary of Anti-Counter Forensics
In this part, we summarize different anti-CF techniques into two categories adversary-aware detectors and generally more secure detectors. Particularly, we compare existing techniques in terms of pros and cons, as shown in Table 12. Table 12 shows that different techniques have their pros and cons in terms of the capability of detection, robustness, efficiency, and so on. Specifically, in adversary-aware detectors, are good to react only against a specific case of attacks. On the other hand, in generally more secure detectors, are more resistant versus CF attacks and then, more difficult to attack, particularly even in the PK scenario.

Concluding remarks
This article reviewed existing ML-based approaches for image manipulation detection in an adversarial setting, such as those based on binary manipulation classification. While a number of these approaches may be efficient in different forensic scenarios, they may be vulnerable or less effective against adversarial attacks. There have also been attempts to design image forensic tools for retrieving information and manipulation detection, as well as CF techniques to defeat ML-based image forensic approaches.
We also observed that anti-CF approaches are generally tailored for specific CF attacks. For example, in the context of the Most Powerful Attacks (MPAs), MPA-aware detectors are designed to enhance the security of ML-based image manipulation against a specific class of attacks. In practice, when the MPA cannot be found analytically, a possible approach is to try to determine the MPAs experimentally, by looking at which attack degrades the performance [33].  When the gradient attack employed on pixel domain against a two-class classifier (2C), they achieved zero percentage of misclassified of adversarial images for different detection tasks such as resizing, median filtering, and CLAHE.
Defense Differential Method [83] Accuracy gain is about 76.30% for random feature size K = 30 when the FGSM adversarial attack applied on the network [35].
In anti-CF, an aware-training technique is considered as a way to improve the robustness of the forensic tools against the JPEG laundering attack. We observed that image forensics tools may have poor resilience against JPEG compression. Compared to more complicated attacks, the JPEG laundering attack is easy to perform and does not require any information about the target detector. With reference to the contrast enhancement (CE) detection, the effectiveness of the JPEG-aware training has been evaluated for both ML (SVM-based) and DL (CNN-based) approaches in the literature [98,35,34]. There have also been attempts to design anti-CF systems that are generally difficult to attack by improving the security of ML-based image manipulation detectors [48], for example using randomization of the feature set [83].
Recent investigations in DL have confirmed that adversarial examples present a certain degree of transferability, especially in the computer vision. Additionally, applying adversarial attacks can be efficient in Limited Knowledge (LK) scenarios. For example, Barni et al., [62] showed that adversarial examples against CNN-based detectors are not transferable between matched and mismatched architectures. Random feature selection (RFS) method is one possible approach to improve the robustness of forensic detectors to targeted attacks, particularly in DL. In CNN, the RFS approach includes the selection of a random feature set derived from the network. Thus, the RFS strategy helps to alleviate the dangerousness of adversarial examples [83].
There remain a number of potential research directions, such as the following.
• Vulnerability and fragility of DL methods against adversarial attacks The reviewed literature shows that most DL techniques in image forensics can easily fool against various adversarial attacks.
Although most of the current works focus on misleading DL on the task of classification, based on the studied literature, we can undoubtedly observe that DL techniques are fragile to adversarial attacks in general. Improving the robustness of the DL-based engines against adversarial learning techniques plays a pivotal role in image forensics. Newly, random selection approaches employes to increase the robustness of the forensic image detectors to targeted attacks. This approach extended based on DL features can help prevent attack transferability by modifying the detector's architecture or retraining the detector with random features. On the other hand, increasing the approach's effectiveness of this approach must be examined against a wide range of adversarial attacks. Moreover, identifying the robustness of the random deep feature selection scenarios when the adversary is aware of the defensive mechanism is another interesting avenue in future research.
• Adversarial samples generalize well The reviewed literature proves that one of the most common features in adversarial examples is that they transfer considerably between similar and different architectures. In contrast, the authors in [62] prove that majority of the different cases, the attacks are not transferable in image forensics particularly between different neural networks. Therefore, additional study is required on the attacker's side to know if and how the transferability can be improved, for example, by improving the attack strength or proposing the most powerful attack. We expect that by increasing the strength of attacks, adversarial attacks can be transferred to the target network even in mismatch networks (see Figure 24). We expect that future work will concentrate more on improving attacks' strength to increase adversarial examples' transferability. Given the complexity of the decision margin learned by the CNNs, controlling the amount of distortion applied by the attacker on the image, is not an easy task. Another direction for future research is related to randomization-based defense when attackers aware of the mechanism, such as feature selection and the architecture. Therefore, we expect that the use of different FC layers or different kernels in SVM can improve the security mechanism on the defender side.
• GAN-CF detection need more investigation In recent years with advancements in DL, several techniques have been extended to create fake image contents, e.g., such as GANs [73,74]. GANs have extensively been utilized to generate synthetic images visually indistinguishable from real ones. Therefore we need to develop mechanisms to differentiate fake from realistic images. Most of the existing methods mainly focus on improving GAN image detection with high detection performance without consideration of anti-forensics. Anti-forensics are also recently utilized for these kinds of detection tools, aiming to attack forensic techniques where the attacker might apply GANs to train a generator to counterfeit forensic traces. To expose these detection method's weaknesses, an attacker can devise a generator anti-forensically to • Defense against backdoor attacks Another new class of attacks against DL architectures is backdoor attacks [92].. Backdoor attacks are in the category of causative attacks when training sets are poisoned by introducing a backdoors signal to a portion of the training files that cause misclassification of a payload during a testing time, which poses a new and realistic threat, particularly in image forensic applications. Although several works have been done, difficulties still exist due to the problems' complexity and forensic applications' wideness. Investigating solutions to enhance the security of ML techniques facing such attacks in image forensics plays an important role. For instance, randomized smoothing is one of the strategies newly considered against backdoor attacks, which was extended to verify robustness versus adversarial attacks but still have limited effectiveness. Another challenges is related to semantic and physical backdoor attacks, and researches are left far behind. Therefore, a better perception of those areas would be an essential step towards defeating the backdoor threat in practice.
• Developing security class of ML Machine learning (ML) techniques have proved remarkable performance in numerous application fields, particularly in image forensics. Traditionally, most of ML methods are trained in a benign setting with identical statistical characteristics in training and testing. Whereas, this assumption usually in ML does not work against adversarial attacks, where some statistical features of the samples can be tampered with by an intelligent adversary. Therefore, one way is to design an ML model in an adversarial setting. The susceptibility of ML techniques in adversarial environments and corresponding countermeasures need to investigate more. Future research on ML security, in particular reference to image forensics, may involve the following aspects. The defense strategy facing adversarial attacks for DL techniques should include more studies. Many studies proved that DNNs are inherently fragile to subtle perturbations that affect output results. Although numerous defense approaches have recently been proposed so far to counter such attacks, we can not find a primary solution to these difficulties. Thus, generating a secure ML/DL under adversarial settings will likely represent a significant challenge for the years to come.
• Deep learning interpretability Interpretability in deep learning-based methods is another hot issue, particularly in image forensic tools. Understanding the nature of the black-box generally is difficult and how the network makes a decision. To clarify more, in pattern recognition, DL can accurately classify cats from dogs, but we can not understand which specific features lead to this decision. This problem also happens in most forensic applications. Hence, tracking DL's reasoning would enable to improve most of the forensic networks in terms of security and robustness against adversarial attacks.
• Multiple adversarial attacks to ML This problem happens in most ML/DL image forensic applications. The challenge considerably different when the ML system has to face multiple adversaries. Based on the synthesization of knowledge (SoK) in the survey, only a few papers have studied models with multiple adversaries. Thus, colluding attacks within Adversarial Image Forensics is a hot issue that can be studied further. .