CloudVaults: integrating trust extensions into system integrity verification for cloud-based environments

. While the rapid evolution of container-based virtualization technologies, emerging as an integral part of cloud-based environments, brings forth several new opportunities for enabling the provision of distributed, mixed-criticality services, it also raises signiﬁcant concerns for their security, resilience, and conﬁguration correctness. In this paper, we present CloudVaults for coping with these challenges: a multi-level security veriﬁcation framework that supports trust aware service graph chains with veriﬁable evidence on the integrity assurance and correctness of the comprised containers. It is a ﬁrst step towards a new frontier of security mechanisms to enable the provision of Conﬁguration Integrity Veriﬁca-tion (CIV), during both load-and run-time, by providing ﬁne-grained measurements in supporting container trust decisions, thus, allowing for a much more eﬀective veriﬁcation towards building a global picture of the entire service graph integrity. We additionally provide and benchmark an open-source implementation of the enhanced attestation schemes.


Introduction
The vision of cloud-based Smart Connectivity Networks (SCNs), comprising multiple edge and fog computing node deployments, is seen today as a key enabler for evolving sectors like automotive, smart factories, smart grids, or healthcare [8,25].Simultaneously, their number is expected to increase significantly with the advent of new mixed-criticality services.To this end, the cloud community is already embracing recent well-known technologies, like Network Functions Virtualization (NFV) and Mobile Edge Computing (MEC) [21] intelligent orchestration.These frameworks are based on the unrestrainable "softwarization" process, which will transform physical infrastructures into distributed data centers with advanced virtualization and software-driven capabilities.They are considered the two key enablers for intelligent edge computing and the cloud to operate in tandem; Virtual Functions (VFs) allow for flexibly customizing cloudbased networks to the needs and peculiarities of mixed-criticality applications and expose them as Service Graph Chains (SGCs) and network slices.
Furthermore, with the advent of the Internet of Things (IoT), we have just begun reaping the benefits of this evolution that, however, also brings several new challenges (or rather makes old unsolved challenges urgent to be tackled with); with security, resilience and configuration correctness being some of the major concerns at both logical extremes of a network.While virtualization offers some security advantages (such as isolation and sandboxing), it has issues such as insecure production system configuration, vulnerabilities inside the images, and vulnerabilities directly linked to various container-based technologies (e.g., Docker, LXC, rkt) [7].The primary existing mechanisms to alleviate such issues leverage the concept of trusted computing [4,5,7,18,23], which addresses the need for verifiable evidence about a system and the integrity of its trusted computing base and, to this end, related specifications provide the foundational concepts such as measured boot and remote attestation.A key component in building such trusted computing systems is a highly secure anchor (either software-or hardware-based) that serves as a Root-of-Trust (RoT) towards providing cryptographic functions, measuring, and reporting the behavior of running software, and storing data securely.Prominent examples include Trusted Execution Environments (TEEs, e.g., TrustZone) [22] and Trusted Platform Modules (TPMs) [13].
Despite recent intensive research efforts towards trust aware containers [7,18], none of the existing mechanisms are sufficient to deal with the security challenges of container-based VFs.Firstly, there is the perceived aspect of the incompleteness of integrity measurements or guarantees, due to the traditional focus of trusted computing on the system boot time or, at most, the load-time of applications, without covering system integrity beyond these stages, during system execution, which is especially crucial for high-availability cloud-based environments.After a VF is deployed, the integrity of its loaded components is ignored.Indeed, while a containerized VF should work correctly (as constructed by the orchestrator) just after it has been deployed, it could start behaving unexpectedly (e.g., modify data in an unauthorized way) if it receives a malformed input from a corrupted module acting on the VF.The assurance that a VF works correctly after loading is known as load-time integrity, while run-time integrity refers to the whole process life-cycle.Secondly, it is imperative to ensure the privacy of the VF (and the underlying host) configuration.One overarching theme of building trust for containers is to leverage IBM's Integrity Measurement Architecture (IMA) [23] that measures the integrity of a designated platform.Since IMA measures all components and records them into a single log, each verifier (Vrf) with access to this log (when validating the integrity of a VF) will also get all configuration information of the prover (Prv).Adversaries benefit from such artifacts and are capable of stealing the information of other users' containers.
Compounding these issues sets the challenge ahead: Can we identify sufficient Configuration Integrity Verification (CIV) schemes that can capture the chains-of-trust, needed for the correct execution of a VF during both load-and run-time, and that allow for inter-and intra-VF attestation without disclosing any information that can infer identifiable characteristics about individual VF configurations?Solutions will, in turn, enable the provision of adequate trust models for assessing the trustworthiness and soundness of the overall SGC.

Contributions:
In this paper, we design and implement CloudVaults, a security verification framework for supporting privacy-and trust-aware SGCs, in lightweight cloud-based environments, with verifiable evidence of the integrity and correctness of the VFs.The solution can be either applied separately to each deployed VF, equipped with a virtual TPM (vTPM) security anchor, or the entire SGC, and it enables CIV of the constructed container(s).Key features provided that extend the state-of-the-art include the: (i ) possibility to distinguish which container is compromised, (ii ) the possibility for low-level fine-grained tracing capability (Attestation by Quote), and (iii ) Secure Zero Touch Provisioning (S-ZTP) capability which allows for inter-and intra-VF attestation without disclosing any VF configuration information (Attestation by Proof).Our proposed solution is scalable, (partially) decentralized, and capable of withstanding even a prolonged siege by a pre-determined attacker as the system can dynamically adapt to its security and trust state.We make an open-source reference implementation of all CloudVaults schemes and protocols and benchmark their performance for VFs that are equipped with either a software-or hardware-TPM 1 .The implementation can be used to develop enhanced attestation schemes further using TPM 2.0 and comparative benchmarking.It should also lower entry barriers for other researchers who want to explore TPM-based Configuration Integrity Verification solutions.Overall, our approach is viable for remedying limitations of existing attestation techniques; nonetheless, there is still a need to overcome other open issues towards a holistic end-to-end security approach.
2 Towards Trust-Aware Service Graph Chains (SGCs) Leveraging cryptographic techniques and Trusted Components (T Cs) towards protecting and proving the authenticity and integrity of computing platforms has been extensively researched.Both integrity and authenticity are two indispensable enablers of trust.Whereas integrity provides evidence about correctness, authenticity provides evidence of provenance.There are two possible avenues towards achieving configuration integrity: either make the configurations themselves immutable or make the hashes of the configurations immutable.The latter approach follows the Trusted Computing Group's (TCG) open integrity standards [24], which recommends the utilization of hardware TPMs for storing an accumulated hash over its Platform Configuration Registers (PCRs).TPMs also inherently provide indisputable evidence of authenticity in the form of signatures over data using securely stored keys (Section 2.1).
As aforementioned, IMA accumulates measurements in a TPM.It extends the principle of measured boot, where components are measured in the order in which they are loaded into the Operating System (OS) using Linux OS kernel functionality.By default, IMA measures the load-time integrity of user-space applications and files read by the root user during runtime.It is based on the Binary-Based Attestation (BBA) scheme proposed by TCG, where measurements and attestation consider hashes of binaries.However, even the smallest change in a binary dramatically changes its hash, making IMA measurements susceptible to grow unwieldy as the number of measured objects increases.Furthermore, the temporal order in which files are accessed or applications are loaded can be highly unpredictable, making it difficult to verify the accumulated measurements.The inherent disadvantage of BBA paradigms is the disclosure of the platform's software and hardware configuration, which is a legitimate privacy concern since an intermediate adversary Adv (or a malicious Vrf) can use this information to infer identifiable characteristics about the platform.
Further, the variety and mutability of software and their configurations make it difficult to evaluate the platform's configuration integrity [4] during runtime.Several architectures extend upon the IMA-BBA paradigm [7,18] to provide integrity verification.DIVE [7] and Container-IMA [18] both incorporate IMA for virtualized Docker containers to enable orchestrators (remotely) determine the runtime integrity of containers in cloud-based environments.DIVE distills the measurements to only present configuration information related to containers of interest, while Container-IMA proposes xor-ing measurements belonging to distinct containers with container secrets to preserve their privacy.Irrespectively, both solutions necessitate the exchange of some identifiable information.
In the same line of research, Property-Based Attestation (PBA) [4,5] schemes map the platform configurations to attestable properties in order to avoid the disclosure of the host configurations altogether.Attesting properties has the advantage that different platforms with different components may have different configurations but still yield the same fulfillment of properties.In particular, PBA gives more flexibility for handling system patches and updates [5], but with the deficiency of detail.[26] presents a PBA-BBA hybrid to the cloud environment where an attestation proxy mediates attestation requests between the prover (Prv) platform and Vrf, such that only the proxy can be aware of the correct configurations of a Prv.It then presents to Vrf only the security property of Prv as the attestation proof, thus, preventing exposure of platform configuration information.The inherent limitation of PBA, however, is that it is only applicable to specific properties (which require accurate identification) and is not directly transferable to reflect changes of mutable configurations.

Solidifying the VFs: Inter-Trustability of Service Function Slices
A combination of these concepts is of great interest to the secure composability of SGCs, encompassing a broad array of mixed-criticality services and applications.In particular, CloudVaults strives to enable orchestration of heterogeneous VFs containing mutable configurations by leveraging the profoundness of BBA while retaining the privacy-centered approach of PBA.In what follows, we elaborate on the inherent functionalities of a TPM that are leveraged by CloudVaults.
Monotonic Counters for Trusted Measurements.Internally, each TPM has several PCRs that can be used for recording irreversible measurements through accumulation, e.g., extending PCR slot i with measurement m, the TPM accumulates: P CR i = hash(P CR i ||m).This is an indispensable property towards the creation of strong and transitive CoT.For instance, to enforce and regulate trustworthiness of the system boot sequence we can require that all components measure and verify their successors by the following recurrence construct [22]: I 0 = true; I i+1 = I i ∧ V i (L i+1 ), where i ≤ n ∧ n ∈ N * , I i denotes the integrity of layer i and V i is the corresponding verification function which compares the hash of its successor with a trusted reference value.For example, as in [6], let us assume that we require the boot sequence: seq sinit, BL(m), OS(m), V S(m), V M (m), AP P (m) , where sinit is the value that the PCR is reset to.If we know that the sequence will yield PCR extensions with the values v 1 , . . ., v n , and all components extend PCR j, then we will trust the chain if and only if (iff ) PCR j = hash(. . .(hash(sinit||hash(v 1 ))||hash(v 2 )) . . .||hash(v n )).
Attestation & Policy-Based Sealing/Binding.Attestation can be either local or remote.Local attestation is based on Attestation Keys (AKs), which are asymmetric key pairs AK = {AK pub , AK priv }.To perform local attestation, we enforce usage restrictions (authorization policies) onto AK priv , such as requiring that PCRs must be in a certain state to permit signing operations, e.g., PCR j (from the example above) actually reflects the accumulation of v 1 , . . ., v n .Thus, using AK priv to sign a nonce chosen by Vrf provides indisputable evidence that the machine state is correct.Remote attestation is delegating the verification of PCR j to Vrf, through TPM quotes comprising a signed data structure of the nonce and the contents of a specified choice of PCRs, which Vrf verifies against trusted reference values.Regardless of the attestation method, Prv must also prove authenticity to Vrf.The TPM contains several key hierarchies, but authenticity is founded specifically in the endorsement hierarchy.The root endorsement seed, from which Endorsement Keys (EKs) are generated, passes irrefutable evidence to the EK in a transitive manner.The credibility of the seed, and hence loaded EKs, is usually based on the trustworthiness of the Original Equipment Provider (OEP), which during manufacturing signs, loads, and later vouches that the seed corresponds to a valid TPM [25].

System and (Adv)ersarial Model
System Model.The considered system (Figure 1) is composed of a virtualized network infrastructure in which the application orchestrator (Orc) spawns and governs a set of heterogeneous VF instances as part of dedicated service graph chains.Each graph is composed of the ordered set of VFs that the service runs to manage better the correct execution of the onboarded (safety-critical) application workloads and guarantee its offered attributes (e.g., reliability, availability, performance).As aforementioned, state-of-the-art software engineering trends are based on the VF microservice concept for achieving high scalability and adequate agility levels [14].In our model, we assume the integration of lightweight virtualization techniques, namely containerization [2], where applications are decomposed into a mesh of cloud-native containerized VFs, each one with specific and "small-scope"-stateless processing objectives, packaged on independent virtual execution environments equipped with highly secure anchors (i.e., vTPMs) that serve as our RoTs.Each deployed VF contains workload configurations, such as its software image, platform configuration information, and other binaries (see Definition 1), which are measured and securely accumulated into the PCRs of the loaded vTPM.Hash of a binary † We further use a prime to denote a reference, e.g., h Conf is a calculated reference to the actual hash of the PCR contents.More formally, the Orc maintains a Service Forwarding Graph (SG), of function chains, defined as SG = {s 1 , s 2 , . . ., s n }, where n ∈ N * .Each service chain comprises a set of deployed VFs, s i = {vf 1 , vf 2 , . . ., vf m }, where m ∈ N * and s i ∈ SG, deployed over the substrate virtualized network.The ownership of the physical resources, over which the secure deployment and placement of these SGs take place, is not of interest.Each vf i ∈ n j=1 SG j is defined as a tuple of the initial form: vf i = (id, vP CR, state, EK pub , AK pub ), where id is the unique VF identifier, vP CR refers to an artificial set of PCRs that reflect the obligatory policy (measurements) that must be enforced in the actual PCRs of the target VF, state denotes whether the VF is considered trusted or not (policyconformant), EK pub and AK pub are the public parts of the EK and AK of the vTPM that is uniquely associated with vf i .
In addition, each VF is equipped with a Runtime Tracer (T rce) for recording the current state of the loaded software binary data (during both boot-up time and system execution) to be then securely accumulated into the PCRs of the hosted vTPM.Tracing techniques are used to collect statistical information, performance analysis, dynamic kernel or application debug information, and general system audits.In dynamic tracing, this can take place without the need for recompilation or reboot.In the context of CloudVaults, a detailed dynamic tracing of the kernel shared libraries, low-level code, etc., and an in-depth investigation of the VF's configuration is performed to detect any cheating attempts or integrity violations.Such a T rce can be realized either as: (i ) a static binary analyzer for extracting hashed binary data measurements (i.e., digests) [1], or (ii ) a general, lightweight tracer with kernel-based code monitoring capabilities based on the use of "execution hooks" (e.g., extended Berkeley Filters) [15].
This process builds on top of the IMA feature [23] and records measurements of the VF's software binary images of interest (as specified in the deployed security/attestation policy) that reflect its state/integrity: these can span from hardware-related properties related to the BIOS/UEFI and kernel information, to dynamic properties such as executable code, structured data and temporary application data (e.g., configuration files, file accesses, kernel module loading).When a measurement is extracted (Section 5.2), a register of the TPM accumulates the digest of the captured event data to protect the integrity and constitutes the basis of the subsequent verification of a VF's trusted state: The trust state is the result of the remote attestation functionalities of CloudVaults (Section 5.2), in which the measurements of the software loaded on a VF is verified either locally (Attestation by Proof) or by the Orc (Attestation by Quote) against reference values that characterize known (and, thus, trusted) software configurations.
Definition 2 (Tracer, T rce).Given an object identifier (see Definition 1), the T rce utility returns (in a secure way) the corresponding object's binary data.
(Adv)ersarial Model.Our in-scope threats include both external attackers who exploit existing vulnerabilities in the VF stack, and insiders such as cloud users and tenant administrators who cause security breaches either by mistake or with malicious intentions.Like most security verification solutions, we trust the Orc responsible for the provision, management, and deployment of the VF Forwarding Graphs.The focus is on detecting threats that can lead to the violation of the specified integrity properties, and attacks by those adversaries who can remove or tamper with dynamic data (e.g., configuration files, logged events, etc.) as these can also be inferred from our integrity analysis: the only cause, a verifier would be aware of, whereby an application can get compromised at run-time is the reading of a malformed datum previously written by a malicious process.Unlike existing schemes, our solution can also cope with adversaries that try to manipulate such dynamic, unstructured data which, together with regular configuration files and network sockets, represent the majority of processes interfaces and, thus, can allow a verifier to determine the integrity and trusted state of a VF with a high degree of confidence.
We assume an Adv that has unrestricted virtual access to the user space of a VF, including oracle access to its attached vTPM.Similar to other attestation architectures, we do not consider availability threats, such as Denial of Service (DoS).Further, the computational capabilities of Adv are restricted to the Dolev-Yao model [9], where Adv cannot break cryptographic primitives (e.g., forging signatures without possessing the correct credentials), but can, nonetheless, perform protocol-level attacks.Note also, that we do not consider a sophisticated Adv that can perform stateless attacks that target a program's control flow [11,20] where the measurement of a binary can remain unchanged even though the software's behavior has been altered.In particular, a residential Adv has the following Capabilities: C-1 Unrestricted passive and active oracle access to the attached vTPM: (passive) Adv can monitor the exchange of commands and responses between the TSS and the vTPM; (active) Adv can unilaterally craft and exchange illicit commands to the vTPM trying to manipulate the CIV process.However, as with any oracle, Adv cannot access the underpinnings and secure structures (e.g., PCRs) of the vTPM.C-2 Unrestricted ability to Create, Read, Update, and Delete (CRUD) VF software binary configurations (by Definition 1).Note that we do not consider attestation of memory contents (objects without a unique system identifier); thus, we do not audit direct accesses to the disks and the memory.

High-Level Security Properties of CIV
In this section, we provide an intuitive description of the security properties our CIV scheme is designed to provide and extract the corresponding axioms (Table 5 in Appendix C) representing the trust properties that must be satisfied by the various components (i.e., VF, vTPM, Orc) involved in the creation and management of SGCs.Such end-to-end definitions for VF soundness and security are then analyzed in Section 6, where we prove how CloudVault's design satisfies them in their entirety.Recall that the focus is on trust-aware SGCs with verifiable evidence on the integrity assurance and correctness of the comprised VFs.Verification of the host Virtual Machine (VM), its kernel, and the entire virtualization infrastructure (NFVI) [17] is beyond the scope of this paper.
Trust is evaluated by (securely) measuring the state (and configuration behavior) of a VF at any given point in time, and then comparing the measured state with the reference (expected) state.Note that the vTPM component of a VF is the trusted element that generates the signatures, certified attestation keys, and quoted PCR values in conjunction with the (potentially untrusted) host.Thus, the Properties that must be achieved are: P-1 VF Configuration Correctness.Both load-time and run-time configurations of a VF (by Definition 1) must adhere to the attestation policies issued by Orc (thus, ensuring load-time and run-time VF integrity).P-2 SGC Trustworthiness.It must, at all times, be possible for the Orc to determine the trustworthiness of the entire SGC.An SGC is trusted if all VFs are attested correctly and have shown verifiable evidence that their configurations comply with the enforced attestation policies.This transfers and extends the sound statements on the configuration security properties of single VFs (by P-1) to statements on the security properties of hierarchical compositions of VFs and SGCs.P-3 Attestation Key Protection.To retain trust in a VF, despite mutable configurations, it must be possible to deploy, during run-time, new (certified) AKs that reflect updates to the configuration policies deployed (i.e., an updated set of vPCRs about the VF's expected configuration).The VF, by leveraging the vTPM, must securely create AKs such that AK priv is never leaked to an Adv so that she cannot forge valid CIV messages (thus, ensuring unforgeability), and must present verifiable evidence that the created AK is "bound" to the newly deployed attestation/configuration policies.P-4 Immutability.The measurement process must be immutable, such that T rce (by Definition 1) always returns the correct (actual) measurements.P-5 Liveness & Controlled Invocation.It is assumed that attestation inquiries reach the local VF attestation agent and that the agent responds with an attestation response within a specified time limit.If a VF fails to respond within the specified time limit, this can be considered as evidence of compromise and, thus, the VF is deemed as untrusted.
Besides the aforementioned core security properties, in some settings, Prv might need to authenticate Vrf s integrity verification requests in order to mitigate potential DoS attacks; e.g., an Adv impersonating the Orc might send a "bogus" configuration policy to a VF representing an incorrect (reference) state.This functionality can be easily provided (and verified) by CloudVaults: In a case where the Orc acts as the Vrf, the respective request (reflecting either a new policy digest or an update measurements request -Section 5), can be signed with its (trusted) certificate so that the target VF can verify its authenticity (note that VFs are employed with Orc's certificate when constructed and deployed over the substrate network).On the other hand, for achieving inter-trusted VF communication, where a VF (acting as the Vrf) tries to attest the correctness of another VF, handling a potential forged request will not have any impact on the state measured by the Prv since this cannot result in the update of the configuration policy (can only be initiated by the Orc).Such a malevolent act will impose some additional performance overhead due to the verification process that will be performed.However, this is negligible as will be seen in Section 7.
Our work provides the missing fine-grained details of the already standardized IMA [23] and fills the perceived gaps of dynamic and runtime remote attestation and configuration integrity verification in a complex software stack as the one met in emerging virtualized environments; from the trusted launch and configuration to the runtime attestation of low-level configuration properties about a VF's integrity and correctness.In Section 6, we provide game-based models for our enhanced CIV scheme satisfying all the above properties.
5 An Architectural Blueprint Towards Unified CIV

High-Level Overview
Our schema provides two specific functionalities, Attestation by Proof and Attestation by Quote (see Figure 2), for enabling the automatic and secure establishment of trust between deployed VFs of a service graph.The attached vTPM authenticates the evidence of the integrity state of the service binary images running inside such containers.Key features provided include the: (i ) possibility to distinguish which container is compromised, (ii ) the possibility for low-level finegrained tracing capability (Attestation by Quote), and (iii ) S-ZTP capability for privacy-preserving attestation (Attestation by Proof).The former is a significant feature because, once a VF is compromised, it can be immediately retracted and replaced by the Orc without affecting the entire SGC, thus, catering to efficient SGC management and flexible slicing [3] making CloudVaults viable for practical cloud-based applications.The latter enables the integrity verification of a designated VF without conveying other VF's information (or unnecessary information of the underlying host) to a remote VF (acting as the Vrf), in case of a malicious Vrf being aware of which components the underlying host and other containers have.This is of paramount importance in emerging smart connectivity networks, leveraging cloud-based capabilities to support safety-critical services with strict security, trust, and privacy requirements [10].
The offered CIV allows to assess and preserve the integrity of the deployed VF's Trusted Computing Base (TCB), at load and during system execution, by leveraging the capabilities of vTPMs while ensuring predictability of the internal vTPM PCR values regardless of the order of loading of applications/processes (inside the VF) and reducing performance impact by minimizing the necessary interactions with the host trusted component.It supports complete, configurable attestation that acquires binary signature chains from different unique registers, enabling advanced tracing capabilities to localize areas of compromise.Both schemes rely on the VF to access a T C (e.g., vTPM) with irreversible PCRs.The privacy-enhanced feature builds on the use of an AK within the T C that can only execute a cryptographic operation if a set of PCRs is in a particular (trusted) state, inferring the correctness of the component.Integration of our Enhanced RA protocols in cloud-based environments, comprising dispersed VFs, is convenient since the exchange of messages can piggyback conventional TLS protocols.CloudVaults also introduces the concept of digest lists to limit the reporting of measured software only to the case of unknown software (not added to the digest list deployed by the Orc).This approach ensures predictable PCR values and reduces the usage of the T C, therefore also the performance impact.
Following a similar workflow to the most prominent IMA-based architectures, Figure 2 presents the information flow of CloudVaults between a Prv and a Vrf: In a nutshell, CloudVaults detects offline and online attacks on mutable files by verifying their hashed digest with a trusted reference measurement extracted from a corresponding virtual PCR on the Orc.Attestation reports produced by CloudVaults can include as much information as required based on the already defined attestation policies (including the configuration properties to be traced).
Attestation policies must be expressive and enforceable and can be dynamically updated by the Orc.After defining proper policies, the Orc can proceed to periodically (or on-demand) attest to the modeled configuration properties representing the current state of the target VF.A VF is trusted if its state (at that time) matches the (already measured) reference state.As each VF is a software component, its hashed digest defines its state.By comparing the hashed digest (at any given time) to the reference (expected) hashed digest of the VF, provided by the Orc, we can determine the VF trustworthiness.

CloudVaults Building Blocks
The core of our schemes (Figure 2) is the manageability of mutable configurations throughout the lifespan of a VF and is accomplished by having the Orc mediating any security-critical updates towards the deployed VFs.Whenever the Orc invokes a periodical or scheduled update, either to determine the trustworthiness of a VF or due to changes in configurations, it proactively determines the update's expected implication by accumulating the artificial vPCR construct of the corresponding VF (Step 1R).The Orc then requests the VF to similarly accumulate its PCRs to reflect potential changes (Steps 2R-3R).This update request contains only the PCR index i that must be updated and a configuration file identifier, β ID , to measure.Upon receiving such update requests, the VF then invokes the T rce to measure the requested file(s) and subsequently invokes T C to extend PCR i with the new measurement.The simple update protocol is depicted in Figure 7 (see Appendix A).Recall that an Adv must not be able to tamper with the measurement process, or the trustworthiness of the entire process will be compromised (Section 3).
Furthermore, as mentioned in Section 2.1, the privacy-preserving attestation (i.e., local attestation) requires the use of specialized signing keys, called attestation keys (AKs), which can be bound to specific PCR contents, hence making an AK operable iff the PCRs reflect the particular PCR state in which the AK is bound to.However, to retain the viability and correctness of such an attestation (despite mutable PCRs), we must create and bind a new attestation key whenever a VF is updated.Since the key creation process is best achieved locally, the Orc requests a VF to create a new attestation key based on the trusted measurements that were artificially accumulated before requesting the VF to update its PCRs.First, the Orc computes an Extended Authorization (EA) policy digest based on the trusted measurements (Step 1L), denoted h Pol , which reflects the trusted state in which the AK must be bound to.The policy digest is then deployed together with a subset of PCRs, I, to which the policy applies.Upon receiving such a request, the VF is responsible for creating the attestation key on the T C (Step 2L).To trust that the policy is enforced and that the state of the VF is conformant to the policy, the VF must present indisputable evidence Fig. 3 presents the underpinnings of the protocol for AK creation, where a VF initiates the process by constructing a "key template" based on the received policy digest.This template dictates the key's fundamental properties, i.e., whether it is a signing key, decryption key, or both, and whether it is restricted (operates solely on TPM-generated objects).The template is passed to T C, which creates an AK as a child key of EK.This process outputs a creation hash h Create and a ticket T , where T is computed with the inclusion of a secret value (Proof) known only by T C, which proves that T C created the AK (Step 2L).The ticket is subsequently passed as an argument to the certif yCreation functionality of the T C, together with AK, to enable AK's certification using EK, which, due to being restricted, requires such indisputable evidence about the provenance of an object.The certificate and its signature are then sent to the Orc for verification (Step 3L-4L).The generated AK is trusted iff the signature over the certificate is verified to be authentic, based on the VF's EK pub .The certificate reflects that the AK was created to require the correct attestation policy to be used for signing operations and that the certificate includes a (public) value called the "magic whose presence proves that the signed object was created internally on T C. Attestation by Quote.The protocol for remote attestation using the TPM quote structure is presented in Fig. 8 (Appendix A).In this protocol, the Orc sends a nonce n (to enforce freshness and prevent replay attacks) and a selection of PCRs to attest, I (Step 4R).The VF subsequently passes these arguments to T C which constructs a quote structure comprising the current values of the chosen PCRs, and signs it with its EK (Step 5R), which as with AK creation, proves that the quote structure is internal to the T C. The quote certificate and signature are then sent to the Orc (Step 6R).The quote and its signature are successfully verified by the Orc (Step 7R) iff they are valid, and if the PCR values correspond to the artificial reference values (vPCR) managed by the Orc.
Attestation by Proof.In the Attestation by Proof protocol (Fig. 4), Orc only sends a fresh nonce n to a VF (Step 5L).If VF presents Sign(n, AK priv ) (Step 6L), where AK is a fresh and verified AK, then this is indisputable evidence that VF is in a trusted state (Step 7L).Note, that both the Attestation by Proof and Quote can only attest to the last known measurement.Thus, both attestation schemes are tightly coupled to run in conjunction with the update of measurements protocol for achieving run-time VF integrity.

Security Analysis
We define four core security games where an Adv (defined in Section 3) tries to manipulate the protocol's building blocks in order to diminish the provided security guarantees on a VF's soundness and integrity.Our models aim to correctly detect alterations made to configurations (see Definition 1) by intermittently attesting their adherence to the reference values (vPCRs) in Orc.We describe the circumstances and constraints required for Adv to win and show why this, in some cases, is impossible under the current set of assumptions.
Game 1 (Update Measurements).Notation: -βID is a non-empty set of unique file identifiers (see Definition 1) provided by Orc -h β denotes the Orc-conformant hash of βID -h β denotes the actual hash of βID, i.e., h β = hash(T rce(βID)) -i denotes the PCR that must be accumulated with the new measurement(s) -φ denotes the Orc-conformant value of PCRi after remeasuring βID; φ = hash(PCRi||h β ) The Game: -I identifies a set of PCRs -hPol denotes the policy for using the key -Acert denotes the creation certificate σcert denotes the signature over Acert -AK is public part of the key created -EKpriv denotes the secret endorsement key The Game:  When the Orc requests a VF to remeasure specific configurations (Game 1 in Figure 5), an Adv must deceive the Orc about non-conformant configurations to hide her presence.Such misleading, requires the Adv to manipulate the local measuring process to extend the T C PCRs with expected (bogus) "good " measurements instead of the actual configurations.Obviously, if T rce is compromised, or the PCR extension of the (correct) measurement (conducted by T rce) is disrupted and never reaches T C but Adv instead manages to feed T C with hashes that reflect "bogus" measurements, then Adv wins Game 1.However, if this were to be possible, then we could never trust the measurements in T C. The enforcement of P-4 and P-5 on a VF overcomes such attacks.P-4 ensures that Adv cannot tamper with the execution of T rce, and can in practice be achieved using more complicated (and resource-heavy) attestation methods, such as Control Flow Attestation (CFA) [15].The latter, P-5, requires that a VF always enforces the LTL invariant given in Eq. ( 1).This invariant states that when a VF receives a measurement update request, Req Orc upd , then the VF must not process further requests that create new attestation keys, Req Orc createAK , since they rely on the correctness of the PCRs.Only when Req Orc upd has been properly processed and the PCRs have been extended with the new measurements as requested by Req Orc upd , then a VF may proceed to process Req Orc createAK requests.Together, these properties guarantees prohibit Adv from ever winning Game 1.
In Game 2, an Adv tries to exploit the AK creation and certification process in such a way that Orc believes that the created key can only be used when the PCR values reflect the correct attestation policy, but Adv can use it at her discretion.This win condition is inherently difficult for Adv to achieve since the Orc requires a fresh and verifiable certificate and a signature over this certificate (Fig. 3).The signature cannot be forged by the Adv since it originates from the T C's secret EK.Furthermore, as described in Section 5.2, the certificate object must be generated by the T C in order to be signable by the restricted EK, evident through the inclusion of the magic header in the certificate.The only option for the Adv is to alter the policy digest during key creation, which will inevitably be discovered by the Orc, either through the actual policy digest in the returned key or if the hash of the key is not the certified name in the certificate.Since the policy digest is unique and strongly linked to the PCR contents, the magic header, and the fact that the EK is secret and restricted, Adv cannot win this game (under current assumptions).σa denotes the signature over the certificate -QCert denotes the quote certificate n denotes the challenge (random number) -EKpriv is the secret endorsement key -I identifies a set of PCRs.
The Game:  In Game 3, an Adv tries to falsely convince the Orc that binaries have not been manipulated by exploiting either the quoting process or building a fraudulent certificate.The certificate comprises the current PCR values and the nonce from Orc.Assuming the accumulated PCRs reflect the Adv's presence, she can try to tamper with the certificate creation process to reflect a forged PCR digest.Unfortunately for the Adv, T C will be reluctant to sign the forged certificate since it did not create it.
Note that the PCRs will only reflect malicious alterations to configurations if an update of the measurements is requested and executed after the Adv has tampered with the configurations.We denote the time of an update as tup, time of compromise as t Adv and time of attestation as tatt.Our assumption (that PCRs reflect Adv's presence) holds (and Adv loses) if tatt > tup > t Adv .If Adv can precisely time the manipulation of binaries such that tup < t Adv < tatt, then the PCRs will not reflect her presence, and Adv will win the game (although will be detected in the next measurements update).This attack is called a Time-of-check to Time-of-use (TOCTOU) [19], which is a disadvantage in the proposed protocol and is discussed in more detail in Appendix B. However, as tup(n) − tup(n + 1) 0 (approaches 0), the disadvantage becomes insignificant since Adv's time window for malevolent behavior becomes very small, but will have an impact on the overall resource consumption of the system.Game 4 shows how an Adv can try to exploit the signing process in order to provide a valid signature over the challenge n using the issued attestation key while having modified binaries.The overall goal is to convince the verifier that no manipulation of binaries has happened.Recall that usage of the key is bound to certain contents of the PCRs.The PCRs reflect the binary states; hence the Adv cannot execute the cryptographic signing operation while having modified the binaries, assuming the registers indeed reflect such modifications.This, of course, is also affected by the TOCTOU attack mentioned earlier, and in this case, with a more severe impact.After every update, a new attestation key has to be deployed, and local reference values have to be updated, taking essential resources from the primary operations of the system.We provide a more thorough analysis of this issue in Appendix B.

Experimental Performance Evaluation
Experimental Setup.Our testbed is deployed on a computer equipped with an Intel(R) Core(TM) i7-8665U CPU @ 1.90-2.11GHzrunning the Windows 10 OS.The main goal of this setup is to evaluate the potential overhead of using a T C that will, in turn, allow us to assess the overall protocol scalability towards providing verifiable VF integrity evidence.Therefore, we have opted out of creating a true scale test environment with separate entities but consider a single binary file containing all components.To evaluate the performance of CloudVaults, we constructed the protocols and tested them against IBM's software TPM V1628 using the IBM TSS [12] V1.5.0.Each experiment (protocol) is performed 1,000 times.Note that since we rely on a software TPM as the RoT, of a VF, we chose to create an attestation primary-key as an alternative to the EK for key storage, which adds a small overhead each time the AK is used.Also, we chose to use an ECC key as the EK instead of an RSA-based EK.Performance Results.Our experiments (Table 2) highlight the efficiency of our protocols.The entire process of creating an AK takes no more than ≈ 17 ms (on average), while including the update of binary measurements still requires less than 20 ms (see Appendix B for more details and a comparison to an HW-based TPM implementation).The enhanced attestation schemes are also efficient (< 12 ms); however, without considering any possible network delay that may be present when communicating the attestation data between the Prv and Vrf.With both supporting routines and attestation schemes being extremely lightweight, we can achieve low-cost, rapid attestation capabilities and provide advanced trust assurance services without consuming many computational resources.Such capabilities ensure trust from the perspective of the Orc and further facilitate bilateral trust assurance (even) between service graphs.As described in Section 6, higher levels of trustworthiness result in the need for more resources.That is why it is imperative for the attestation protocols to be lightweight enough without, however, impeding on their accuracy and correctness.To better demonstrate the achieved effectiveness, we use Eq. ( 2) (Appendix B) to determine how fast we can detect binary manipulation.In the worst-case scenario, where an Adv tampers with a binary just after an update, she will remain undetected for at most 293.40 ms, if we utilize as little as 20% of the CPU time.
The ease of operating CloudVault's protocols, including their efficiency, makes the framework highly applicable to be integrated into large-scale networks.While we did not take processing-or network-delay into consideration and only use the AK once, the experiments show that the time of detection is in the order of seconds.In Fig. 10 (Appendix B), we further see that even with 10% utilization, we can still detect a change after ≈ 1 second, making it extremely difficult for an Adv to manipulate CloudVaults.

Conclusions
In this paper, we proposed CloudVaults, a multi-level security verification framework for supporting trust aware SGCs with verifiable evidence on the integrity assurance and correctness of the comprised containers: from the trusted launch and configuration to the run-time attestation of low-level configuration properties.Based on our analysis, we described how a VF achieves privacy-preserving integrity correctness and how to utilize vPCRs for binary data integrity with a virtual-based RoT.Our prototype and the evaluation results demonstrate that our architecture can satisfy the privacy, security, and efficiency requirements.Furthermore, by considering the salient characteristics of remote attestation, we identified several open research challenges.We believe that if these challenges are tackled now while container-based CIV is still at an early stage, this emerging security mechanism can reach its full potential.Implementation Note.Writing protocols in terms of TPM calls requires reading and understanding the TPM 2.0 specification and this makes TPM development challenging and causes a high-barrier of entry.While the TPM 2.0 specification was designed to be easily maintainable, it is nevertheless challenging to read mainly due to its sheer size.It consists of over 1400 pages split into four parts which not only cover the core specifications, but also numerous errata covering the continuous development of the TPM specification.Therefore, a particular TPM will be based on the core specification and all of the relevant errata which it implements.HW-based TPM Timings.The timings for executing the individual TPM commands of the CIV protocols are presented in Tables 3 and 4, and are performed using IBM's software (SW) TPM V1628 and the Infineon (HW) TPM 2.0 chip.The mean time is calculated from repeating all experiments 1,000 times for the SW-TPM and 100 times for the HW-TPM.The values reflect the time between executing a command in the IBM TSS [12] V1.5.0 and until receiving a response.The SW-TPM timings are much faster than those when using the HW-TPM.Even though a hardware TPM has some degree of hardware accelerated cryptography, it still cannot measure itself with a modern CPU, and is not designed to do so.Applying Eq. ( 2) on the HW-TPM yields a time of detection as t d = (2 • 734.89 ms + 417.49ms (1 + 1) + 6.09 ms)/0.20 = 11.5 s, which is indeed larger than that for the SW-TPM.However, these values are somewhat misleading since the host CPU's utilization does not have any effect on the HW-TPM as it executes the operations on the hardware chip itself.By evaluating the TPM command execution and application timings, we can see that the most time-consuming operations are those executed on the TPM, which is why the impact of the CPU utilization using an HW-TPM is significantly lower.Removing this constraint from Eq. ( 2) gives us t d = 2.310 s (excluding the host times, such as verification, nonce generation, etc.).Additionally, the "create primary key" function is extremely time-consuming, which is why it might be useful to load this AK from NV storage.

Fig. 10 :
Fig. 10: (a) Changing the number of attestations each key has to do and its impact on the time of detection (20% utilization) and (b) shows how different utilization of resources impact the time of detection with one AK use.

Table 1 :
Notation used Atmp Key template hConf † Expected configuration (PCR hash) hPol Policy digest based on hConf hCreate Key creation hash, w.T C state and parent key T † Creation ticket proving origin of creation hash Acert Key creation certificate QCert Quote Certificate h β †

1 .
Goal: Adv conceals non-conformant configurations by deceiving the remeasurement process.2. Setup: Adv in VF with C-1 and C-2. 3. Challenge: An update request, Req Orc upd , containing the pair {βID, i} is received from Orc and is accessible by Adv, where βID identifies a file object blemished by Adv. 4. Response: N/A. 5. Win condition: Adv wins the game iff PCRi = φ, although h β = h β .

1 .
Goal: Adv returns a verifiable key that Adv can use at own discretion.2. Setup: Adv in VF with C-1 and C-2. 3. Challenge: Orc provides Adv with PCR Selection I and a policy digest hPol.4. Response: Adv responds with creation certificate Acert, signature σcert, and the certified public key AK, constructed by Adv or T C. 5. Win condition: Adv wins the game iff the key created inside the T C contains a different policy digest than provided, while simultaneously providing a valid and verifiable certificate, public key, and signature from the restricted signing key EKpriv.

1 .
Goal: Adv presents valid signature and certificate with PCR values that hide Adv presence.2. Setup: Adv in VF with C-1 and C-2. 3. Challenge: Orc challenges Adv with n and PCR Selection I. 4. Response: Adv responds with certificate QCert and signature σa, constructed by Adv or T C. 5. Win condition: Adv wins iff Orc can verify σa over QCert (containing n) signed by EKpriv and the accumulated digest in QCert matches Orc's accumulated digest from vPCR.Adv provides verifiable signature over challenge, despite of modified binaries.2. Setup: Adv in VF with C-1 and C-2 (see Section 3), and AK has been deployed.3. Challenge: Orc (or secondary VF) challenges Adv with n. 4. Response: Adv responds with σa, either constructed by Adv or T C. 5. Win condition: Adv wins the game iff Orc (or secondary VF) can verify σa being a signature over n signed by AKpriv.

Table 2 :
Timings of CloudVault's protocols (time in ms).Note that the hashing is done without any secure hashing schemes and might be slower in practice.

Table 3 :
Mean time (in ms) of using SW-and HW-TPM for updating measurements and creating a new AK.

Table 4 :
Mean time (in ms) of using SW-and HW-TPM for Attestation by Quote and Proof.