A Flexible Inter-locale Virtual Cloud For Nearly Real-time Big Data Applications

The rapid growth of emerging applications such as social network analysis and ecosystem monitoring has led to an explosion in the amount of data being generated. On the other hand, the increased capabilities of cloud computing afford more flexible computing resources to achieve better performance. However, it is still difficult to determine how to provision resources to fit large scale applications across different locales. It is essential not only to be able to manage the greater volumes of data but also to ensure that the QoS (quality of service) requirements of processing are reliably satisfied. This paper presents a Cloud engine that can provision a flexible network transparent virtual Cloud. In this virtual Cloud, complex virtual infrastructure is distributed over multiple data centres or Cloud providers. The Cloud engine can help application developers to customise and provision virtual infrastructure for applications with critical performance constraints that may influence how and where application components should be hosted. First, we introduce the capacity gap between such applications and the Cloud. We then describe the cloud engine along with its key technologies, including transparent network connectivity and standardised multi-level infrastructure descriptions, as well as its applicability to particular scenarios. Following these descriptions, we discuss and demonstrate how to apply the Cloud engine to help in a specific big data application use case and satisfy its QoS constraints.