Chapter 1: Turning Open Government Data into Public Value: Testing the COPS Framework for the Co-Creation of OGD-Driven Public Services

: This chapter aims to demonstrate and understand how open government data can generate public value by allowing any actor to co-create an open government data-driven public service. The chapter takes a holistic approach to understanding open government data-driven co-creation and follows a content-con-text-process approach for the framework development. The framework proposes a public service co-creation cycle based around the ideas of agile and lean development that should lead to increased usage of open government data. The co-creation cycle is made up of four parts: co-initiation, co-design, co-implementation, and co-evaluation. To test the propositions put forth by the framework, a multi-case study was conducted on five different pilot projects that aimed to use open government data in the co-creation of new public services. The pilots were conducted at different levels of government and across different public domains. The results of the study seem to support the propositions outlined by the framework, though it also emerged that the pilots that engaged in co-implementation had higher levels of user engagement and satisfaction with the service; this warrants future empirical research.


X.1 Introduction
Open Government Data (OGD) initiatives are springing up across the globe at every level of government [1]. Due to this trend, OGD is seen as an increasingly powerful source of value, both economic [2] and public [3]. In simple terms, public value can be understood as the total societal value that is shared by all actors in society [4]. More specifically, public value has been defined through five key dimensions: direct user value, indirect value to wider societal groups, social value (i.e. support to social interaction and cohesion), value to environ ment and sustainability, and political or democratic value [5]. However, public value is not something automatically appears when OGD is made available. The concept of public value has a strong connection to the idea of co-creation and the co-production of services -so, public value can be seen as something that is co-created by different stakeholders such as service providers and service users through the process of mutual interaction and co-production [6] Though it is not the only touted benefit of OGD, see: [2,3,7], the creation of new and innovative services that create public value does appear to be one of the greatest potentials associated with the OGD movement [8]. When OGD is made available to the public, all societal stakeholders, whether governmental or external, can come up with ideas for using data to solve existing problems and needs, and then co-create these solutions together [8,9]. However, the question how and by which process OGD can be turned into services that provide public value is generally still under researched [3]. This chapter aims to address this research gap by examining how governments at different levels can co-create public value fro m OGD initiatives.
The chapter posits that one of the main ways of turning OGD into public value is for public administrations to encourage and engage in the co -creation of OGDdriven public services. The definition of a co-created OGD-driven public service has two core components: public service and co-creation. When talking about public services, the authors have adopted the definition recently put forth in [4], which states that any service, developed by any stakeholder, that creates public value may be viewed as a public service, regardless of the role that the public sector plays in it. The second component, co-creation, may be defined as the involvement of outside, non-typical, stakeholders in the initiation, design, implementation, and evaluation of public services [10]. Thus, we come to the definition of a co-created OGDdriven public service as a public service that exploits OGD to create public value and has been co-created amongst different stakeholders. This chapter will present a framework that outlines what exactly co-created OGD-driven public services are and how these services come into being. The framework takes a holistic approach and looks at how services are developed, but also acknowledges the importance of contextual factors on the OGD ecosystem.
The development of this framework began as part of the OpenGovIntelligence project 1 , a European Union funded research and innovation action that aimed to explore how OGD may be used to drive the co-creation of new public services. In addition to developing a theoretical framework, the project also involved the implementation of OGD pilot projects. Based on a multi-case study of these pilots, this chapter will discuss the practical applicability of the framework. The pilots represent a variety of different OGD maturity levels, are conducted at different levels of government, and are creating services in a wide variety of sectors. However, they are also similar in that all pilot projects aim to develop new services by exploitin g OGD and engaging in co-creation with different stakeholders. This case study research will help provide insight into the utility of the proposed framework, and will allow for a foundational level of understanding to be constructed of co -created OGD-driven public services.
The chapter is structured in the following way. First, a framework for understanding how OGD may be turned into co-created public services is presented, based on the current state-of-the-art when it comes to OGD, co-creation, and cocreated OGD-driven public services. Once this is done, the case study methodology, research design, and potential limitations will be discussed. This is followed by a description of the six pilot projects where special attention is paid to the unique operational environment of each pilot. The final step will be to apply the framewo rk to the pilots, discuss the results and implications, and conclude with proposals for future research.

X.2 COPS (Co-created OGD-driven public services) framework
The ideas proposed within this chapter represent a shift from a traditional understanding of public services and public service delivery. In order to better understand this change, and to acknowledge the intricacies and complexities that accompany the change, the proposed framework takes a holistic view on the co -creation of OGD-driven public services and follows a content-context-process (CCP) approach (see: [11,12]). In practice, this means the framework looks at the content first (what exactly is a co-created OGD-driven public service); second, the context (what are the drivers and barriers, the operating environment, agents, etc.); third, the process (what must happen in order for the concept of co-created OGD-driven public services to be realized).
The combination of the content, context, and process come together to generate a new picture of the co-created OGD-driven public service system. The framewo rk that results will provide a clearer understanding of how co-creation of OGD-driv en public services occurs and will provide insight into how governments can drive or initiate the co-creation of OGD-driven public services. The framework draws on ideas and theories from public administration and management research (the ideas of co-creation and co-production), e-Government and Information Systems (Open Government Data), strategic management and computer science (agile development), and additionally is influenced by trending ideas in the current startup ecosystem (Minimum Viable Product (MVP) and Lean Development).

X.2.1 Content: Co-created OGD-driven Public Services
The concept of 'public service' has been defined in a wide variety of ways. For example, in Estonia, a public service is defined as something that the state or government provides at the expense of the state for the benefit of society [13]. This is similar to how many academics, scholars of public administration, and government officials across the world perceive and understand public services . However, this is beginning to change [14,15]. In the United Kingdom, there has been an increased interest in the idea of "open public services". This concept aims to open up public service provision to a wide range of providers, decentralize the public service provision process, and divest control of public services to the service users thus increasing their ability to choose and customize their services to fit their needs [16]. In a similar spirit, a recent report by the European Commission (EC) titled "A Vision for Public Services" (2013) proposed, "public services are services offered to the general public and/or in the public interest, with the main purpose of developing public value […] The future of government is less and less in the hands of governments alone. Technology has empowered ordinary citizens by offering them a way to make their voices heard" [4]. This framework adopts a similar understanding of public services, and it is also an understanding that many in the current scholarly debate are beginning to move towards (see: [15,17,18]).
Traditionally, public service providing organizations attempted to understand what issues society was facing, and then aimed to create or draft some sort of service to address the needs of society; this was often done without consulting the intended recipients and the provided service may or may not produce the intended effect [19]. In this model, services are delivered in a top down manner, with citizen as customer, dependent on the government, and often given little role to play in the design and implementation of the service [19]. However, due to the development of ICTs and open and participatory governance models this approach seems to be outdated. The new understanding of public services aims to bring the provision of public services into today's modern age and many public service organizations are beginning to experiment with new ways of public service provision.
In line with the definition provided by the EC, the idea of "co -creation" has begun to flourish in academic and governmental discourse. In essence, co -creation is about stakeholders from a wide variety of groups who come work together to "cocreate" something new. This means that government agencies may be working with private individuals, NGOs, companies, or other stakeholders; the government agency may or may not be the one steering the design and implementation of the service. It is believed that a public service delivery process steeped in co-creation may lead to increased efficiency and effectiveness of public s ervices [20][21][22], it is part of the wider open government movement [23], and is a necessary part of the current movement to bring citizens into a more collaborative relationship with government [23,24].
The term co-creation is tightly linked to Elinor Ostrom's concept of co-production, but also has strong roots in service management theory [20]. Recently, the upand-coming public administration paradigm of New Public Governance has embraced co-creation as an imperative part of its platform [18]. Though there are many different understandings and definitions of co-creation, many tend to view it as a multi-faceted process with different stages, each with their own unique way of involving stakeholders in the relevant "co-" process. One such classification was put forth by [25] who saw co-creation as a four stage process consisting of co-design, co-decision, co-implementation, and co-evaluation. This is similar to the classification provided by [21], which states that co-creators can co-discover problems, coinitiate solutions, co-design the services, and co-implement the newly developed services. There is also increased interest in digitally enabled co-creation, which has been discussed by [26,27] In tandem with co-creation, governments have also begun to realize that traditional waterfall model based approaches may not be as effective as other project management styles, such as agile development. Thus, public service organizations have also begun to adopt agile development methodology and ideas into their internal processes [28]. When talking about agile development, the following definition is adopted: Agile development focuses on being able to adapt quickly to changes by following an 'agile' approach based on multiple sprints made up of four main stages: plan, build, test, release [29,30]. The agile development cycle allows projects to be designed and implemented faster and become more responsive to changes, such as customer preference or environmental factors. In the public service design context, an agile development approach is more conducive for co -creation than the traditional waterfall model. The reason for this is that an agile approach allows for input and feedback to be provided on the service at multiple points so that it can be integrated and acted upon in future sprints, whereas in the traditional approach this is largely not possible.
Though the adoption of agile development by public sector organizations can indeed be beneficial, another innovation should accompany it in order to produce the biggest value. This accompanying idea is that of lean development and the minimum viable product, MVP; in the public sector context, the product may be un derstood as the public service. Lean development, as proposed by Eric Ries in his book "The Lean Startup", implements a development cycle that follows a build -measurelearn structure ( Figure 1 shows this cycle, adapted for the public service context).

Figure 1. Lean Development Cycle. Source: Authors
The core idea behind the lean development cycle is that the organization should be able to learn as quickly as possible about whether or not their product will be well received (in the public sector context, the product is the public service). As part of the cycle, an MVP is developed in an agile manner, and then presented to the customers (in the public sector context, customers are the service users). Once the MVP has been built and presented, the build-measure-learn cycle begins and the process of 'validated learning' starts; validated learning may be understood as the process of understanding whether or not you are building the correct o r desired service for the user [31].
As public service organizations aim to become more agile, reimagine how they deliver public services, and embrace ideas like lean development or co -creation, a new revolution is also occurring, a data revolution: data sets are bigger, our processing power is stronger, and data is now becoming increasingly open and available to everyone [32][33][34]. The idea of OGD finds its roots in the broader open government movement with the aim of promoting transparency and accountability [23,35]. In addition to these aims, and due to the growing availability of OGD sets, there has been an increased effort to define and understand what OGD is, and what the potential uses, benefits, drivers, and barriers of OGD may be [3,[36][37][38][39]. When discussing OGD, it is generally agreed upon that in order to be classified as OGD it must be free to reuse and redistribute by anyone, be human understandable, be government organization generated, and preferably come in a machine readable format [3,37,40]. Though studies have aimed to understand and present the potential benefits of OGD (see: [3,7,41]), it has also been found that often times the availabilit y of data does not necessarily translate to new benefits [3]. One way that OGD may provide public value is by exploiting it and creating new and innovative services on top of it [10,42,9,43]. Due to widespread availability of OGD and data analytics tools/languages, such as R or Python, any stakeholder is able to beg in to analyze OGD and/or build services that rely on or utilize OGD [8,9]. This has drastic implications for the public service delivery process as, now, a stakeholder can find their own answers or create value on their own, rather than having to rely on a government agency to provide the answer or build a service that may or may not solve the stakeholder's initial problem.
Though the use of OGD in the creation of new public services is an interesting area of study, in order for this phenomenon to occur at a broader level, a framewo rk for understanding and analyzing the process is needed. Putting together all of the aforementioned changes that are currently ongoing in the public sector domain, the idea of a co-created OGD-driven public service begins to emerge. The new paradigm that accompanies this idea includes the following:  A new understanding of public services that is based on the idea of public value and where any actor is able to participate and take the lead in the cocreation of services that create public value.  The traditional top-down waterfall based approach to public service development is outdated and in today's networked and IT-oriented society needs to be updated to reflect the current paradigm.  The co-creation of new public services is likely to benefit from a process based around the concepts of agile and lean development methodologies .  It may be possible to improve the effectiveness of public services by creating and releasing an initial MVP.
 As OGD may be exploited by any actor with sufficient technical knowledge, OGD can be used to co-create innovative services that create public value.
This leads us to the first proposition of our proposed framework:

Proposition 1 -OGD can be turned into public value through the co-creation of OGD-driven public services
To provide an initial visual aid that demonstrates how the main components of the framework (OGD, co-creation, public services, agile development, MVP) fit together, Figure 2 has been created. Figure 2 shows that co-creation is an iterative process based around the ideas of lean and agile development and it takes in OGD. The initial result of this iterative process is the MVP; once the MVP is released, the iterative co-creation process continues and the result is a co-created OGD-driv en public service.

X.2.2 Context: Operating Environment, Drivers, and Barriers
The co-creation of an OGD-driven public service should be thought of as taking place within a system (see: [8,55]). The system is made up of the different agents (such as public sector organizations, citizens, etc.) that take part in the process of co-creation and of different environmental factors that support or create impediments to the functioning of the system. Public sector innovation and e-government literature often emphasizes the importance of the technological, social, organizational, administrative, cultural and political context as a source of driv ers and barriers to technological innovations in the public sector (see, for example [44][45][46]). Thus, a second proposition can be put forward:

Proposition 2 -The results of the co-creation system are influenced by the contextual environment
In the context of OGD and co-creation, the immediate environment includes the interests and abilities of the stakeholders involved in service co-creation, the data infrastructures for OGD publication and exploitation, as well as the legal, political and organizational context in which OGD-driven co-creation takes place.
Stakeholders. The very idea of co-creation suggests the involvement of more than one stakeholder group in the creation of public services. The groups often mentioned in the context of OGD and co-creation are public administrations, citizens or citizen organizations, businesses, and academia (see, for example, [49]). These in turn can consist of various different sub-groups with different needs, interests, skills and positions, and hence different roles in the co-creation process. The new conception of 'public services' proposed above sets no limitations to the role that any of these groups can take in data-driven co-creation: all of them can act as initiators of new data-driven services, or as partners and co-creators of these services. This, however, not only presumes the existence of supporting infrastructures but also a favorable cultural environment for data sharing and cross -sectoral collaboration.
Stakeholders' interests, values, perceptions and capabilities have been found to play a crucial role in co-creation. Stakeholder perceptions can be both the key driver as well as a major barrier to the supply of OGD and the use of OGD for service cocreation [37]. Since open data is often perceived as lacking tangible benefits while costing a lot, there is resistance in many organizations to making their data open. Similarly, the benefits of co-creation are not well understood, which manifests in the administrators' lack of openness to the idea of co-creation [50]. This is further complicated by a widespread lack of necessary skills to open up data and make use of open data in innovative ways among all stakeholder groups. On the other hand, stakeholders' beliefs, priorities, preferences, skills and actions can act as a powerful driver of OGD -for example, visionary policy-makers and administrators can act as innovation champions promoting the publication of OGD, and grassroots groups and individual innovators can express demand for open data and demonstrate the possibilities to re-use OGD in innovative ways [37].

Data infrastructures.
To ensure the quality of data and easy access to datasets, infrastructures are needed that support the publication and re-use of open data. Some of the important elements of such infrastructures include [37]:  A central free open data portal where local and national governments could publish their data. Such open data portals should have the ability to host data, sign-post to remote data, cache datasets, and provide tools for data transformation across various formats or via various web services requests.  If necessary, data infrastructure legislation should be adopted to regulate the maintenance and access to data assets, and the rights, roles and responsibilities connected to that.  Providing APIs. Implementation of the "API First" policy means that governments should prioritize providing good APIs along with open data (rather than make external stakeholders download data dumps) to increase the reliability of data and facilitate the reuse of open government data by external stakeholders. However, the mere existence of an OGD portal is not a sufficient driver in itself. One of the best examples of this is the National OGD portal in the USA and the municipal OGD portal of the City of Chicago. While the national portal has a large amount of data, many datasets go unused and it could be argued that the level of public value it aimed to create has not yet manifested. Meanwhile, in Chicago, there is an active civic hacking scene and new public-value creating innovative applications are being created on a seemingly constant basis (see [8], [47]). One of the primary reasons for this is familiarity with the data [48] and the relevance of the data to those who are exploiting it [8,47].

Legal Environment. The supply of OGD is also constrained by legal issues
around Intellectual Property Rights, personal data protection, security, data sharing and choosing appropriate licenses. For example, personal data protection regulations sometimes prevent the government from releasing datasets that would otherwise be interesting for service innovators. Although this problem can generally be overcome by data aggregation into larger statistical datasets, this is not always a solution if the data concerns very small groups of people. Privacy -related concerns seem to have layers: one is connected to the actual regulations and the other with the way they are perceived and interpreted by public sector organizations [37]. The misunderstandings that some public officials might have about privacy and identityrelated information might also impel them to be overly cautious about publishing any data rather than figuring out ways to publish data without privacy violation risks. Similarly, limited awareness about existing data licenses can be a barrier to making data open and reusable.
Generally, the contemporary public procurement culture and contracting legislation are rooted into the short-term efficiency idea (see, e.g., [51]) that also restricts the use of agile development methods and implementation of risky projects by the government. Therefore, the provision of OGD may be a quicker solution than waiting for legislation and culture to change. The availability of OGD gives non -governmental stakeholders the ability to take on this risk, create services in an agile, responsive, adaptable manner, and constantly learn and improve instead of failing at a larger scale.
According to our previous research [37], the main drivers of OGD publication can be seen in favorable data licensing and copyright regulations that are compatible with open data goals, public interest and new business models, as well as the awareness of public officials of personal data protection regulations. It has also been suggested to introduce a national-level legal obligation for government institutions to make public sector data open by default and qualify public grant submissions and public tenders against open data.
Policies. Policies hold a considerable potential to further drive OGD innovationa potential which still needs to be unlocked. Based on [37,52], European open data policy, in particular the Directive on the re-use of public sector information (PSI Directive) and its open-by-default principle, is seen as a good example of how policy can drive OGD publication at the national level. However, there sometimes seems to be an implementation gap -the obligation is there but it is not enforced by Member States. Another critical driver is seen in the presence of a holistic approach to open data policies, i.e. regarding open data as part of a broader open government policy and supporting this by a combination of legal, policy and technical measures. In addition to that, OGD provision and use can be supported by data standardization policies, which should be tackled at a cross -border level, benchmarks with other countries to create peer pressure, and funding of different forms of collaboration (cross-border, cross-sectoral, inter-organizational) to enable learning and enhance cooperation between data producers and data users.
Organizational and Administrative Factors. The organizational context of the public sector is a frequently cited impediment to implementing innovative technologies and practices. For example, rigid organizational structures, inertia, organizational silos, lack of collaboration, lack of incentives for innovation, risk avoidance, lacking innovation capabilities, lack of innovation leadership, and resource constraints in the public sector are often seen as barriers to innovation and co-creation (see, for example [53]). In the context of OGD-driven co-creation, similar barriers have been noted, including incompatible organizational routines and processes; lack of feedback loops between government and citizens; lack of openness to the idea of open data and open processes, lack of trust and innovative culture; lack of political priority; lack of adequate resources [37]. In addition to that, open data innovation is also hindered by existing proprietary business models and the fact that many public organizations make part of their revenue by selling key data [37].
At the same time, a favorable organizational context can also drive innovationsome of the important drivers are ICT literacy, slack resources, active innovation leadership, strong political support, inter-institutional collaboration, etc. [54]. In the case of collaboration and co-creation with non-governmental stakeholders, additional factors become important, such as the openness of the organizational culture towards citizen input (Freeman and Quirke, 2013). Some of the key ways to mitigate the existing organizational barriers to OGD-driven co-creation are as follows [37]: remodeling the existing processes for public service production t o a co-creationbased approach; development of new business models on top of OGD; Capable change management; capacity-building in public sector and non-governmental organizations regarding digital skills, OGD, data management and service co -creation.

X.2.3 Process
As explained above, the process of co-creating an OGD-driven public service takes place within a system. The anticipated result of the system functioning is a new co-created OGD-driven public service. However, in order for this anticipated result to emerge, a fundamental understanding of the process is needed. Thus, this section focuses on providing an overview of this process.
When talking about co-created OGD-driven public services, what is really being talked about is a new, radical, and innovative approach towards designing, implementing, and understanding public services. The most critical piece of this new understanding is the new conceptualization of a public service as any service that creates public value. The reason for this assigned impo rtance is that, if traditional understandings are utilized, it would not be possible for any stakeholder (such as a citizen or a company) to take the lead in the public service creation process. This leads us to the third proposition of the framework:

Proposition 3 -Any stakeholder (even individual citizens)
is able to take the lead in the public service creation process.
As was mentioned in the content sub-section of this chapter, there has been increased movement towards agile and lean development of public services [28,56,57]. Though the literature most often looks at how these development strategies are implemented at the governmental level in a top-down manner, there are clear benefits for the uptake of agile and lean development in the context of co -created OGDdriven public services. As co-created OGD-driven public services have the explicit goal of bringing multiple stakeholders together to create a new service, it is paramount that communication, feedback, and learning takes place, and it takes place often; the combination of agile and lean development makes sure this communication happens. In practice, the idea is that an initial service should be released as an MVP, this MVP is then tested and functionality is either continued and/or changed as needed, depending on the feedback received. This starts a constant sprint -like cycle where, after the initial development, the service is continuously improved and developed until it reaches completion; the fourth proposition of the framewo rk emerges from this idea: Proposition 4 -There should be an initial release of a public service at the earliest possible stage as an MVP so that the process of validated learning and development may be started as quickly as possible.
When thinking about the concept of co-creation as a four-step process as proposed previously in this chapter it is possible to see a potential bridge between the co-creation cycle and agile development cycle. Table 1 shows the relationship between service producers and service consumers, their motivation for engaging in co-creation, and the relationship between agile development and the respective co -creation step (note the four steps in the proposed cycle have been adapted from [21,25,58] It is interesting to point out that in this process of OGD-driven co-creation, the service producer and consumer are asking the same motivational questions as, in co-creation, the service producer and consumer are not clearly delineated roles and one stakeholder will often play the role of both producer and consumer. Based off this mapping between agile development and co-creation, Figure 3 was drafted and represents a new agile development-based co-creation public service framework. In this framework, any stakeholder is able to take the lead and initiate, design, implement, and evaluate a new public service. This represents a large shift from the traditional top-down approaches of public service delivery. Furthermore, it should also be noted that Figure 3 denotes an iterative cycle; OGD plays a catalytic role and enables co-initiation, but once the service has been co-initiated the co-creators rapidly iterate through the co-design, co-implement, and co-evaluation stages until the co-created OGD-driven public service is finished. In the model shown in Figure 3, any actor is able to provide feedback at any stage of the cycle and during any iteration (be it the first or the last).

Figure 3. Co-Created OGD-Driven Public Service Cycle. Source: Authors
The final two propositions of the presented framework relate to Figure 3: Proposition 5 -Input of the service consumer should be sought and given consideration at all stages of public service development.
Proposition 6 -The public service should be able to be change and/or respond in a fast and efficient manner based on received feedback from the service consumer.
In order to understand the different actions that may take place during each 'co-'stage, Table 2 has been prepared; this figure lists potential actions and contributions that fit into each stage, but it does not claim to be an exhaustive list.

Co-creation Stage
Co-creator Contribution Co-Initiation  Problem and needs identification  Idea generation  User story generation  Target group identification Co-Design  Service user interface mockups  Service process design  Participation in design workshops Co-Implementation  Contributing code online through a medium such as GitHub  Helping to gather or clean data  Engaging other stakeholders in the co-creation process Co-Evaluation  Providing feedback on the service  Raising issues to service developers  Reporting on data quality Table 2

. Co-creation Stages and Actions. Source: Authors
The framework presented in this section represents a dramatic reimagining of how public services are conceptualized, built, and implemented. By following the framework, governments should be able to create public value from their open government data sets. Furthermore, an agile development approach that puts a heavy emphasis on citizen involvement in the co-creation of services should lead towards a transition of citizen as customer to citizen as collaborator (see [59]); this is one of the biggest potential benefits of this framework. Ultimately, the framework has the following goals:  Transition towards "new public services" [14] that place public value and citizens at the center of public services.  Conceptualize the idea of a co-created OGD-driven public service.  Redesign the process associated with public service provision.  Provide an easy-to-follow process to turn OGD into public services.  Empower communities by allowing them to become public service producers.  Encourage active civic involvement by all stakeholders due to decreasing the barriers for participating in service co-creation.  Allow governments to harness the power of citizen participation and collaboration.
It can thus be assumed that if the proposed process is followed, and appropriate attention is given to the factors identified in the context section, governments should see increased levels of public value creation. This occurs due to the design and creation of new public services that come into existence because of the availability of OGD. Furthermore, if public service providers acknowledge that other stakeholders are able to take the lead in public service creation, there should be an increase in new and innovative approaches for solving citizens ' problems and issues.

X.3 Research Design
In order to understand to what extent the underlying assumptions of this conceptual framework hold true in practice, we conducted an exploratory multiple case study of five pilot initiatives that had been undertaken within the Horizon2020 funded OpenGovIntelligence project to demonstrate how OGD could be used to create public value. A case study strategy was chosen because of our goal of investigating the phenomenon in its real-life context [60]. Due to the novelty of the concept of cocreated OGD-driven public services, not much information about such initiatives is yet available in literature; hence, the OpenGovIntelligence pilots provided a valuable opportunity to access detailed information about the content, context and process of such initiatives, while enabling to study OGD-driven public service co-creation in different circumstances. The selection of the pilot locations for the project was based on the following criteria: 1) The pilots needed to address a relevant and current societal issue or problem; 2) The pilots were required to use OGD to address the selected issue with the end goal of increasing public value, i.e. every location that wanted to be selected as a pilot location had to be able to demonstrate a problem that could be fixed through the exploitation of OGD; 3) The pilot cases were also chosen based on their variety in terms of country context, representation of different levels of government (national, regional, local), and different policy domains.
This resulted in the selection of six pilot cases covering six European countries -Belgium, Estonia, Greece, Ireland, Lithuania and the UK. In addition to these countries' differences in terms of the overall political and public administration system, they also differed in their level of OGD maturity, involving early adopters (e.g. the UK) as well as laggards (e.g. Estonia, Lithuania). The chosen pilots also represented different levels of government and involved different policy/service domains. Although the OpenGovIntelligence project involved six pilots, one pilot (Greece) was omitted from our study due to its focus on using data to improve public sector internal decision-making rather than co-creating a public service. We therefore ended up investigating five pilots in more detail.
The aim of our study was to empirically validate our understanding of the elements and processes that make up the OGD-driven public service system. More precisely, we asked the following specific questions about each pilot in order to map them against the key propositions of the framework:

Proposition 1
Did the use of OGD enable to address the indicated societal need or issue?
Did the use of OGD allow for a public service to be cocreated? What kind of public value was (or will likely be) provided by the pilots?

Context
Proposition 2 What drivers enabled or supported the process?
What barriers hindered or constrained the process? Did the application of the framework enable stakeholders to overcome some of the previously existing barriers?

Proposition 3 Proposition 4 Proposition 5 Proposition 6
How was the service creation process implemented? What steps did it involve?
What stakeholders were involved in the process? Did the service creation take place by way of co-creation between different stakeholders, including those not typically involved in public service provision? What method was used for service development? Did the use of agile development enable to speed up the development process? Did it support co-creation?
Did the pilots produce an MVP ? Did the provision of an MVP allow for a quicker collection and integration of feedback from service users and stakeholders?

Table 3. Questions Asked During Pilot Analysis. Source: Authors
The following data sources were used to collect information about the cases: The research approach has some clear limitations , which should be kept in mind when interpreting the results of the study. First, the involvement of the authors in the Estonian pilot may raise the question of possible bias in interpreting the results. In order to minimize the risk of bias, we paid careful attention to triangulation between multiple sources of data [60], being open to contrary findings [60] and comparing evidence from the Estonian pilot with the other pilots where the authors had no role. Second, due to the small and unrepresentative sample, the results of our study are not statistically generalizable to other cases. However, findings from a multiple case study do allow for analytical generalizations to be made about theoretical propositions [60]. Third, due to the limitations of the research design, the case study only allowed us to develop a basic understanding of the ability of the framework to describe and guide the co-creation of OGD-driven public services, while not allowing conclusions to be drawn about the effect of particular contextual variables (e.g. government level) on the results of the pilots. Last but not least, as the pilots are still ongoing at the time of writing this chapter, our conclusions are based on the results of the first phases of the pilots and are thus preliminary, whereas the long-term results of the pilots still remain to be studied.

X.4 Pilots
The pilots conducted within the scope of the OpenGovIntelligence project aimed to demonstrate how OGD could lead to the generation of public value by following an innovative public service co-creation process. What is unique about these pilots is that they provided researchers direct access to validate and test the proposed framework across a wide variety of scenarios:  Pilots were conducted at different levels of government (regional, municipal, and national).  Pilots had different goals (such as increased transparency and imp roved decision-making).  Pilots aimed to create services that could be used by different user groups (internal to the implementing organization, external to the implementin g organization, or both internally and externally focused).  Pilots took place within a wide variety of contextual domains varying from unemployment and social policy to maritime search and rescue.
As every pilot has its own specific use case and context, each pilot will be described in more detail with special attention being paid to the specific problem it aims to address, the solution to address the problem, and the OGD that is utilized by the solution to address the problem.

Pilot Country One -Belgium. The Belgian pilot has been initiated and driven
by the Flemish Department of Environment, Nature, and Energy. This organization aims to ensure a healthy and sustainable environment. One part of this goal is to ensure that there was a clear understanding of industrial pollutant emissions. To achieve this understanding, the Flemish government has required companies operating within Flanders that wish to emit polluting substance to apply for a permit and then to report yearly on their pollutant emissions. This data has been collected since 2004, and the government is now working on opening up the data so that it may be used and analyzed by companies, the general public, and by public sector organizations. The opening up of this data was done in response to complaints from companies who believed that if there were such stringent reporting requirements, that the data that was collected should be opened so that some value could be generated from the reporting. This pilot believes that by making the pollutant information pub-lic, new analytical dashboards can be built that allow for easy and efficient monitoring of emission trends. Furthermore, stakeholders with an interest in ensuring adherence to environmental regulations are able to monitor companies and check for any irregularities or violations that may occur. This pilot acts as a proof-of-concept for the Flemish government, demonstrating how opening up data may allow for the creation of new and innovative public services that increase cooperation and communication between society, government, and private sectors. The initial se rvice takes the form of an online dashboard that allows individuals to view pollution on a map, compare across regions, timescales, and conduct other forms of statistical analysis on the data.

Pilot Country Two -Estonia. The Estonian pilot is being implemented by two different organizations, Tallinn University of Technology and The Estonian
Ministry of Economic Affairs and communications. The Estonian pilot aims to fight information asymmetry in the Tallinn real estate market by provid ing users an easy way to access data relating to real estate. In Estonia, much of the information relating to real estate is either closed or not easy to find/access. This means that when an individual navigated to an online real estate portal, they may be able to find out the size of the apartment, its condition, and the price, but nothing else in relation to the environment, safety, or other environmental factors. The pilot aims to remedy this by aiding in the opening up of new data sets and by bringing t ogether relevant datasets into a single-point-of-access portal. The Tallinn real estate portal allows anyone to search for an address and find all data that may be relevant for a given address. For example, the user is able to find information about crime in the area, car crashes nearby, school locations, public transport, and information about the building. The initial version of the pilot proposed to use and bring together 11 different data sources. The initial target group for this pilot is foreigners wh o are moving to Tallinn, Estonia and may be trying to find out more information about where they are moving to so that they can make an informed decision about where to live in Tallinn. The pilot has been built in a completely open source manner, utilizin g open source technology, and has encouraged and sought out outside input throughout the entire design and implementation of the pilot. The Estonian pilot was coinitiated by foreigners living in Tallinn, the University, and the Ministry of Economic Affairs. Though the pilot's initial focus is foreigners moving to Tallinn, it has the potential to provide value to government officials, real estate agents, investors, and other stakeholders who may be interested in real estate data. The benefits of this pilot include increased timesaving for those trying to find real estate information, a decreased administrative burden, and increased transparency in the real estate sector leading to fairer prices.

Pilot Country Three -Ireland. In Ireland, the pilot is being led by the Irish
Marine Institute, which is a state agency with a mandate to research and innovate within the marine sector. The Marine Institute maintains a large amount of data that is available in an open and linked format, but there were some issues when it came to accessing and creating value from this data. The Marine Institute found that three areas where could use OGD to generate new and innovative public services: search and rescue, renewable wave energy, and maritime tourism. The primary focus of the pilot was to collect and make data available in real time. In regards to the first scenario, search and rescue, the availability of quality and easily accessible real time data could aid rescuers by providing them information about the current size of waves, wind speed, or other conditions where a rescue needed to take place. In the second use case, open data related to the waves would allow researchers to plan and optimize the locations to test new solutions for creating energy from the movement of waves. The final use case aimed to provide value to those who wish to engage in leisure activities on the water such as boating or swimming. With real time data available, stakeholders would be able to make informed decisions about the safety of their activity at a certain time or be able to be better prepared for a situation they may encounter such as a storm. The Maritime institute collects data from multiple sources such as weather stations, buoys on the water, and other statistical reports and then make its available and open in real time. This data can be both queried statistically using a language such as SPARQL, but it is also possible to view in real time data related to specific areas of the Irish coastline on a map. Overall, the pilot aims to provide value across multiple sectors in the form of safer and more effective search and rescue operations, allow for increases in informed decision -making, decrease administrative burden, and also allow for new and innovative services to be built on top of the data.

Pilot Country Four -Lithuania.
In Lithuania, a pilot project is being coordinated by Enterprise Lithuania that aims to increase business and entrepreneurship within the city of Vilnius. At the start, there was no information in regards to the opportunities that were available to businesses in the city of Vilnius, so, in order to remedy this, a portal has been created that allows for easy visualization of data to allow businesses to make more informed decisions. The portal was initially conceptualized by Enterprise Lithuania (a state agency), but the user experience and the design has been generated and created in cooperation between Enterprise Lithuania and local business owners and entrepreneurs. In order to build and implement this portal, data was opened and made available to the public; this data largely dealt with potential markets, active businesses, demands, and current distribution of businesses across different sectors. The portal foresaw potential entrepreneurs and businesses thinking about entering the Lithuanian market as the initial target group, but it is also likely that citizens with an interest in accountability and fair businesses practices will engage with and analyze the data made available on the port al.

Pilot Country Five -United Kingdom.
The pilot that is taking place within the United Kingdom is being run and organized by Trafford Council, which is a government organization responsible for the area of Trafford in the area of Greater Manchester. In the UK there is a problem when it comes to the distribution and usage of Job Centre Plus locations; there are over 800 locations maintained by the Department of Work and Pensions. These locations provide a place for citizens to claim their work benefits, gain assistance with interviewing, receive job training, receive help in applying for jobs, and generally are supposed to aid those who are having difficulty with any aspect of obtaining employment. The location of these centers is being reviewed, as it is believed the current system is not as efficient as it may be. The pilot being organized by the Trafford Council aims to understand the location of these centers within their area and also look at how they are being utilized, by who, when, and for what reasons. Using OGD sets relating to Job Centre Plus locations, worklessness, poverty, and other related datasets a pilot program has been constructed that will allow for policy makers to gain a better understanding of the usage of each centre and also see which areas are currently over or under served. The pilot has actively sought and engaged input from outside stakeholders and met with other government decision makers, private sector companies, and mangers of Job Center Plus locations to discuss what data is needed and how it should be presented. Overall, this pilot creates a new dashboard that allows government officials to make decisions that are more informed due to increased availability of data and easy to understand visualizations. Thus, citizens and users of Job Centre services are also to benefit as locations and services provided will be optimized based on information from this pilot.
To provide a summary of the different pilots and their domains, Table 4 shows the level of government where the pilot was initiated at (either municipal, region al, or national), the problem domain of the service, the type of end user for the service (internal to the public sector, external, or both), and the overall motivational goal of the pilot. There was a mix of pilots across government levels (2 from regional, 2 from municipal, and 1 from national) which operated in five different domains.
Pilot Level Domain Type of users Goal Belgium

X.5 Results and findings
The proposed framework latches onto the idea that digital technologies, such as OGD, have the potential to transform public services. The framework aims to provide a new way of understanding, designing, and implementing these services. It is stated that the availability of OGD has the potential to act as a catalyst for co -creation, and that a public service creation process that embraces co -creation, agile development, and lean development may drive the creation of new and innovative services that provide public value. Therefore, the ultimate test of the value of this framework is to what extent the application of the proposed process allows to create public value from OGD and successfully engage different stakeholders in this cocreation process.
The framework put forth six propositions with regard to the phenomenon of cocreated OGD-driven public services (some of these propositions have been previously published in [10]). Next, the findings of the empirical study will be presented as regards each proposition, with the goal of understanding whether the application of the framework allowed for public value to be successfully co-created from OGD in each pilot.

Proposition 1 -OGD can be turned into public value through the co-creation of OGD-dri ven public services
Due to the focus of the OpenGovIntelligence project on the exploitation of OGD to create public value, the core component elements of a co -created OGD-driv en public service were present in all pilots. All exploited OGD to address some societal need, and all applied a co-creation process to create concrete services whereby the value of data could be released. The findings from the first rounds of pilot evaluation conducted within the project showed that all of them had a public value-oriented goal: the Belgian pilot aimed to increase transparency in the domain of environmental pollution; the Estonian pilot aimed to increase transparency and reduce information asymmetry in the real estate field; the Irish pilot aimed to improve marit ime search and rescue services; the Lithuanian pilot aimed to help businesses make better decisions on where to locate their activities; the UK pilot aimed to improve public decision-making and public services targeted to tackling worklessness. In order to achieve the goal, all pilots engaged different organizations and stakeholder groups in a process of co-creating the respective services.

Proposition 2 -The results of the co-creation system are influenced by the contextual environment
All pilots demonstrated the importance of context as a source of drivers and barriers for the co-creation process. For example, for the pilots in Estonia, and Lithuania, data availability and quality turned out to be major challenges due to a low level of OGD maturity. However, in the other pilot countries where a higher level of OGD maturity existed, these challenges did not present themselves to the same extent. In some pilot countries, the organizational beliefs also posed a major challenge to the co-creation of new OGD-driven public services. In the case of Estonia there was minimal government support due to the belief that only a government should provide services, whereas in the United Kingdom organizations are actively pushing for more user involvement and co-creation. Though all of the pilot countries strove to involve outside stakeholders, getting individuals to participate in the co -creation of the service was difficult. This appears to be linked to the fact that four pilots were co-initiated at a government level and, therefore, perhaps there was not much interest from citizens, private sector, or non-profits.
The pilots' experience also suggests that the application of an agile and collaborative service development process effectively helped bypass some of the main barriers to the use of OGD for public value creation. For example, the Estonian case demonstrated that if government organizations lack the interest and capacity to initiate OGD-driven services, such services can well be initiated and created by nongovernmental stakeholders such as a group of university students and researchers.

Proposition 3 -Any stakeholder (even individual citizens) is able to take the lead in the public service creation process
A large majority of the pilots were initiated by stakeholders in the governmental sector, the exception being Estonia. In the Estonian pilot, a foreign researcher living in Estonia noticed that there was a serious need for more information to be provided on the real estate market, that this data existed, and that th e data was not easily accessible. This, then, led to an Estonian university taking the lead role in initiating the co-creation of an OGD-driven web application to address this need. The role of the public sector partners in this case was limited to providing data for the application and participating in a co-design workshop where user stories were created and the functionalities of the application were defined. Proposition 4 -There should be an initial release of a public service at the earliest possible stage as an MVP so that the process of validated learning and development may be started as quickly as possible.
The use of agile and lean development principles varied widely across the pilots. In Estonia and the United Kingdom, an MVP service was develop ed, released, and then improved over multiple iterations. In these two cases the development was all done in an open source manner, an initial service was released, and the end -users of the service were consulted and their feedback integrated into each successive cycle of development, thus leading to more personalized services.
In the other cases (Belgium, Lithuania, and Ireland) user input was also sought, but development was not conducted in an agile manner and the code was not open. Though a new service has been created in all pilot cases, the two pilots in Estonia and the United Kingdom are the easiest to evaluate and monitor as all improvements, issues, and comments have been raised and are visible online; whereas in the closed development cycles these issues are not transparent.
Proposition 5 -Input of the service consumer should be sought and given consideration at all stages of public service development.
All pilots involved a sort of a co-creation element (co-initiation, co-design, coimplementation, or co-evaluation), but this manifested itself in different ways in the different contexts. In order to discuss each of these "co-"steps in detail, definitions are provided for each step:  Co-initiation occurs when service users play a critical role in getting service producers to create a new service or response.  Co-design occurs when users and producers of a service interact with each other and both are able to influence the design and direction of the service.  Co-implementation may be understood as the process in which input from service users is required for the service to function or where the service user plays a critical role in building or implementing the service.  Co-evaluation occurs when users of a service provide feedback and this feedback is available and used by other service users or service providers.
In an ideal world, all four steps would be followed to have a truly "co -created" public service. However, what was made clear by studying the pilo ts is that is difficult for all four of these elements to take place, and quite often only two or three steps are actually put into practice by the service producer. In the case of the pilots, all had elements of co-design, only three pilots had strong elements of co-implementation (Estonia, United Kingdom, and Ireland), and all pilots had elements of co-evaluation. Thus, it does appear that of the four proposed stages, co -implementation is the hardest to implement in practice. It is unclear why co -implementatio n occurred easier in some pilot countries compared to others as where it did occur and where it did not occur contained an equal mix of contextual background, and it is not possible to identify what caused this.
When looking at how each "co-"stage manifested across the pilots there were many different approaches. For example, all pilots conducted user workshops at the co-design stage where stakeholders from different groups were brought together to discuss the use case, service design, and direction/focus of the new service. However, at the co-implementation stage, two separate approaches were used. In the Irish pilot, users were able to upload their own data, refine and improve available data, and will soon be able to use sensors to help provide and g ather data for the service producer. On the other hand, the United Kingdom and Estonian pilots aimed to involve outside stakeholders in the implementation of the service by making the code open source and encouraging active participation from service users in the actual coding of the pilot. Additionally, the Estonian pilot worked with civic hackers to help improve some internal functions of the pilot. Interestingly, the pilots that engaged in co-implementation also had the strongest levels of user engagement and interaction. This is interesting as it does seem to suggest that in terms of the four "co-"stages, it may be the most important when it comes to the co -creation of public value and facilitating active co-creation of a new service.
Proposition 6 -The public service should be able to be change and/or respond in a fast and efficient manner based on received feedback from the service consumer.
In regards to the last proposition, services that went through more iterations (United Kingdom and Estonia), tended to be more open, more responsive, and have a higher usage rate than the services that did not follow an iterative development cycle. There are a few potential reasons for this. Firstly, it seems to be the case that when users are involved throughout the co-creation process they are more attached and engaged with the service, and thus feel a sense of ownership and will continue to engage with the service over time. Secondly, services that start with an initial MVP launch and improve overtime simply provide more opportunities for engagement with other co-creators, and more opportunities for engagement with lower barriers would understandably lead to higher levels of interaction between service user and service provider. Thirdly, services developed in this manner are able to transition the direction of the service quickly, so if initial users of the MVP point out issues they can be dealt with immediately rather than later on in the process where changes may not be possible. Thus, services that are co-created in an iterative manner are more responsive and in-tune with the users' needs, which helps drive efficiency of the service, provide higher levels of public value, and also appears to drive higher levels of user engagement and empowerment.

X.6 Conclusion
The growing availability of open government data is widely held to open up new ways of creating public and commercial value. However, not much is yet known about how exactly public value can be extracted from OGD. This chapter argued that one of the prominent ways of turning data into value for citizens and society is the co-creation of public services. Such services are 'public' not in the traditional sense of being provided or funded by public administrations but in the sense of contributing to public value and common good. If OGD is made available at a broad scale, any stakeholder that has the interest, ideas and skills can take the lead in building OGD-driven services that address some sort of societal need or add value to citizens' lives in different ways. This chapter proposed a framework that explains the concept of co-created OGD-driven public services (COPS) and put forward a collaborative process for the creation of such services, while taking into account the effect of various drivers and barriers in the broader context. The core ideas of the COPS framework were formulated as six key propositions:  Proposition 1 -OGD can be turned into public value through the co-creation of OGD-driven public services.  Proposition 2 -The results of the co-creation system are influenced by the contextual environment.
 Proposition 3 -Any stakeholder (even individual citizens) is able to take the lead in the public service creation process.  Proposition 4 -There should be an initial release of a public service at the earliest possible stage as an MVP so that the process of validated learning and development may be started as quickly as possible.  Proposition 5 -Input of the service consumer should be sought and given consideration at all stages of public service development.  Proposition 6 -The public service should be able to be change and/or respond in a fast and efficient manner based on received feedback from the service consumer.
In order to explore the ability of the conceptual framework to describe and g uide the co-creation of OGD-driven services in practice, we conducted a multiple case study of five pilots that were implemented in five different countries and five different domains in the framework of a European project. The analysis of the pilots supported most propositions of the framework:  All pilots used OGD to co-create public value through the creation or improvement of public services .  The pilots also point to the effects of context -for example, the lack of OGD availability turned out to be the main barrier for pilots that were implemented in countries with a low level of OGD maturity.  Some pilots met challenges related to engaging public sector stakeholders but due to a new conceptualization of 'public service' and the application of a co-creation approach, the lack of participation of public sector organizations could be overcome by non-governmental stakeholders taking the lead in developing the service.  As regards the fourth proposition, the use of agile and lean development principles varied widely across the pilots, but following the agile and lean development cycle seemed to yield better results in terms of speeding up the cycle of service creation,.  All pilots also involved elements of co-creation, mostly in the form of codesign and co-evaluation. Since user input was sought and utilized in all pilots, it is difficult to evaluate to what extent the application of a co-creation approach may lead to more effective services compared to cases where cocreation is not used. Nevertheless, it is possible to say that the pilots that had higher levels of user participation and feedback tended to be viewed as more effective thus giving some credence to the idea that increased user engagement throughout the "co-" cycle leads to services that are more in tune with the service users' needs. Interestingly, the three pilots that had strong elements of co-implementation with service users and other stakeholders had stronger levels of user engagement and interaction compared to other pilots.  Finally, as regards the last proposition, the services that went through more iterations (United Kingdom and Estonia) tended to be more open, more responsive, and have a higher usage rate than the services that did not follow a development cycle that was able to respond fast to user needs.
Based on these cases, it appears that following the proposed framework can lead to the co-creation of OGD-driven public services and that the framework is applicable across a wide range of domains, problems, and environments. In regards to co-creation, it is interesting to see that despite the many barriers associated with this concept in literature, co-creation did occur in every pilot. One reason for this may be due to the breaking up of the co-creation process into four stages. This four-stage development approach provides more opportunities for stakeholders to contribute to the co-creation and lowered barriers to participate compared to other traditional approaches. Interestingly, the co-implementation stage seems to be the most important stage of the cycle. Thus, it follows that those who wish to benefit from OGDdriven co-creation should consider putting a large emphasis on this stage.
The analysis of the pilots allows us to conclude that the core concepts of the framework are useful and applicable in many different contexts. At the same time, several elements of the framework, in particular the process, still require further empirical exploration in order to understand how the process may be refined to achieve the best results in terms of creating public value from data. More research is also needed on the positive and negative effects of different contextual factors on the co-creation of OGD-driven public services. While context was only superficially touched upon in this chapter, we see broader environmental drivers and barriers such as political interest, attitudes to co-creation and availability of OGD as important elements of the OGD-driven public service ecosystem. Lastly, future research might also examine how people's familiarity with the data and proximity to the issues that are being solved with the help of data affect citizen engagement in the co-creation of OGD-driven services. For example, although the methodological limitations of our study did not allow us to explore this hypothesis, previous research (e.g. [8], [48]) seems to hint that OGD at the local and municipal level may induce more active citizen engagement than OGD at the national level, thus possibly making the local government the most important arena where public value can be co-created.