Realising the potenti al of technology in education Government of Sierra Leone Education Data Hub : A User Research Report

Report on the Government of Sierra Leone Education Data Hub

3.6.2. "What could be the worst thing the Data Hub could evolve into, as far as you're concerned?" 17   Table 4. Initial categories used for recruitment 37 4. The top features and content requests by users and potential users are downloadable raw data, and background information on the data itself and the education system in Sierra Leone. Some wanted more granularity. User feedback also indicated a need to improve how the data is visualised.

5.
A lack of maintenance and support, unreliable data, political manipulation of the data and an unclear product vision were participants' greatest fears. They also 1 For information about the Education Data Hub, see: https://educationdatahub.dsti.gov.sl/education 2 For further information on the Directorate, see: https://www.dsti.gov.sl/ 3 For further information on the Ministry, see: https://mbsse.gov.sl/ 4 For further information on the EdTech Hub, see: https://edtechhub.org/

Introduction
The Education Data Hub is an online platform developed in 2019, by Sierra Leone's Directorate of Science, Technology and Innovation (DSTI) in partnership with the Ministry of Basic and Senior Secondary Education (MBSSE). The objective of the Education Data Hub is to improve the quality of education and ensure all students in Sierra Leone are learning, by making education data easily accessible for use in planning, accountability and monitoring, resource allocation, and spending decisions. The Data Hub also gives citizens access to the information they need to hold educators and the government accountable.
The Education Data Hub currently connects the Annual School Census and the National Examination Results. It serves as a tool to support research, planning, policy and practice decisions, and the evaluation of interventions across schools.
In May 2020, the EdTech Hub partnered with MBSSE and DSTI to conduct user research 5 for the Education Data Hub, working with the digital transformation consultancy Public Digital . Prior to updating the Data Hub with 2019 data, and expanding its functionality, 6 the MBSSE and DSTI and digital visualisation partner Beyond Words Studios (BWS) 7 sought to better understand how Education Data Hub users currently are accessing and using Hub data and what more is needed to improve their ability to find and use this data in decision making. Key questions framing the research conducted by Public Digital include: • How is the Education Data Hub used in making decisions and solving problems by different stakeholders? • How can the Education Data Hub be improved to maximise the use of data in support of student learning and accountability?
This report is the output of this user research and is structured as follows: • Methodology: how research participants were identified and recruited, limitations and user groups.
• Research findings: insights from the user research, plus the ideas and strategic questions they generate.
• Beyond the user research findings: what we found from Amazon Web Services (AWS) analytics, and what we expected to find in the user research that we did not.
• Recommended next steps: product development and strategy recommendations, based on both the user research findings and the research team's own experience.

Methodology
This is a summary of the detailed methodology section in the appendix.
Based on initial research and conversations with the MBSSE and DSTI team members, we identified and focused our recruitment on three main groups of users: 1. Civil society, including: parents of children of or nearing school age; private organisations in the education sector; academics; school principals / headmasters and mistresses 2. Civil servants, including: Individuals in charge of assessing the quality of the education delivered (e.g., school inspectorate representatives), as well as those in charge of school infrastructure (buildings and facilities), and those in charge of global education policies and their evaluation.
3. NGOs and IGOs: organisations of all sizes, including a large intergovernmental organisation (UNICEF), a medium-sized organisation with offices in multiple countries, and a small Sierra Leonean NGO.
We conducted 19 interviews between 11 and 24 June 2020. The interviews were conducted remotely, using Zoom and WhatsApp. On average, the interviews took 40 minutes. All insights about the use of the Data Hub in this report come from the user research interviews.

Research findings
This section on research findings describes the insights that we gathered from our interviews, including who uses the Data Hub, how it is used, perceptions about the Data Hub, and how it could evolve in the future.

Who uses the Data Hub today?
The six interviewees who disclosed that they have currently or recently used the Data Hub were: • From two small international NGOs • Working for an education provider in Sierra Leone • Working for an education software provider in Sierra Leone • A PhD student at an American university

• A civil servant in the Teacher Services Commission
We learnt that the Data Hub is not yet well known. Seven out of nineteen people we interviewed had either not heard of it or had not used it, as shown in the table below. Among the twelve people who had heard of the Data Hub before, half had never used it.
In the absence of sufficient, reliable, web analytics or traffic data, however, we cannot determine if the sample of users that we interviewed is representative of all current Data Hub visitors.
Due to our small sample size, we have grouped both current, recent and past users of the Data Hub together (Table x). Of the 12 interviewees who know about the Data Hub, six have not yet had an opportunity to use it.

How interviewees currently relate to the Data Hub
This section describes how current users interact with the Data Hub.

How the interviewees learned about the Data Hub
Participants mentioned learning about the Data Hub in various ways, including through a connection at the DSTI or MBSSE and discovering it while looking for other information on the MBSSE website. Several people mentioned hearing about it from communications shared by Minister Sengeh, either on WhatsApp groups or radio shows.
However, awareness is not always understanding, and the term 'Hub' seemed to confuse a few people who thought it was a networking platform or a physical location.

Why the interviewees use the Data Hub
Among the people who are currently using it, the reasons cited for using the Hub were: • To inform on-the-ground projects at the planning stage, including funding applications (e.g., target specific areas / schools based on below-average exam pass rates; estimate how many students can be reached through a project; identify and get in touch with local actors).
• To forward information to third parties who don't know where to ask or look for, e.g., enrollment data.
• To inform academic research.
• For private organisations in the education sector: ○ to estimate market size (e.g., based on the number and the typology of schools) and inform business strategy; ○ to build a pipeline of commercial leads, for example retrieving the contact details of school principals.
• To forecast future needs, for example: "what schools will need improvements in 3 years to be able to welcome all the children" Users from NGOs and IGOs stated that they always complemented the information they accessed from the Data Hub with: • raw data, obtained through a request to DSTI or MBSSE; • additional data collected through surveys, partners or on-the-ground research they were conducting: " The hub brings the picture to mind but for my work, I then use raw data to get to the details" World Bank datasets, USAID's Demographic Health Survey (DHS), AfroBarometer, UNESCO Institute for Statistics and OECD were cited many times as complementary sources.

Why the Hub is not being used even when interviewees are aware of it
Of the 12 interviewees who had heard about the Data Hub before we engaged with them, 6 had not used it for the following reasons: • They did not have access to a computer.
• They had no identified use case for the Data Hub over the time period they had known about the Data Hub.
• They preferred using raw data from the MBSSE, which made their analysis simpler and easier.
One participant explicitly noted that they could only access the internet on a mobile phone and therefore couldn't access the Data Hub.
Other participants who knew about the Hub but were not users, included: • A consultant with an organisation that specialises in cleaning data for future uploads onto the Data Hub. They are involved in assessing, providing and / or cleaning data for various organisations like the MBSSE or DSTI, and have experience in seeing how their work is then featured on websites like the Education Data Hub.
• Consultants who are in-between government contracts, with no current need for accessing the Data Hub, but likely to find it useful in a future project. Civil servants often seek raw data that they can manipulate for their own analysis.

Strategic questions:
• Should the Data Hub aim to replace other internal storage locations of education data, and become the unique point of access for the latest education data?
• Should the Data Hub establish itself as a first point of access for high-level data insights. What problem would that help to solve (which team is being currently asked these questions, what would they do if they spent less time answering them)?
• Is it reasonable to expect civil servants to go online for their internal • data needs, instead of getting it on physical supports (e.g., USB, CD) that don't require access to the internet?
It is important to keep in mind that the Data Hub doesn't include dynamic data feeds that could trigger return visits, nor is it regularly updated. Most users seem to have specific immediate needs that don't encourage regular or repeat visits, particularly when they can access the raw data. The Data Hub in its current state doesn't encourage regular visits

Strategic questions:
• What would be the value of regular visits that don't correspond to an immediate or specific need?
• Who would benefit from that value?
• What would be the cost of creating and maintaining content that would entice such visits?

What Data Hub users are asking for
This section describes the information that users are looking for when they come to the Data Hub.

Downloadable raw data
All users asked for the ability to download the raw data, for conducting further analysis with their own data analytics software solution or to compare it to other data. They appreciate the graphs which give them a first-pass sense-check of the data in a user-friendly way.

Background information on the data and system
Participants, regardless of their current use of the Data Hub, also asked for more information about how the data was collected, processed and analysed, in particular when it came to the graphs displayed on the Data Hub: "I need to know where the data came from and how the graph was constructed" "you need to know exactly what the chart means" The provision of explanations was also seen as an opportunity to learn:

"I'd love to develop skills in that area [data analysis], so it would help me"
Many interviewees were unfamiliar with Sierra Leone's education system. They expressed an interest in having more information about the education system, such as a definition of "approved schools", or equivalences for the national exams.

User perceptions of the graphs
Several interviewees described the graphs as sophisticated, sometimes unclear, and likely too complex for untrained people or those with limited comprehension and data analysis skills.

"The World Bank website or USAID's DHS allow you to create quite simple graphs, I use those functions quite a bit -if I want to know trends in school enrollment across countries, I can do that."
While users value features that enable them to quickly visualise data as illustrated in the participant quoted above, they also asked for more information about the methodology used to create the graphs and some of the terminology.
Interviewees with a background in economics -from academia and NGOs, offered that some of the graphs could be misinterpreted as suggesting causality where there is only correlation.

Figure 3. Insight and strategic questions 3
Insight: The existing graphs are appreciated by users but were described as overly complicated and a likely barrier to lay users.

Strategic questions:
• What is the best way to visually present data for the various audiences of the Data Hub?
• How can the needs and abilities of highly educated and mathematically savvy users be balanced with users who have less capacity to manipulate data, and understand sophisticated graphs and charts?
• How can the key data points that are provided as "ready baked" graphs include more informative narrative on their importance and value?

User expectations and feedback on the Data Hub
Users shared that they expect the Data Hub to: • Deliver insights into the Sierra Leonean education system so that they can plan projects and policies (increase efficiency for users).
• Provide an entry point for government information about the education system, and signpost to further information. For example, this could be through links to relevant pages or policy documents on the MBSSE's website.
Overall, user experiences with the Data Hub as captured in the direct quotes below, were very positive. "Everything has proven essential". "The search function, really cool, works with minimal internal skills." "The visualisation engine, I think it's very powerful and allows to make sense of very large data sets " Currently, very little official data on education in Sierra Leone can be readily accessed online. User sentiments are that "anything is better than nothing" and that the Data Hub is a modern and efficient way to deliver information.

User reflections on how the Data Hub could evolve
We probed users with the following two questions to gain insight into their perceptions of the positive and negative ways in which the Data Hub could evolve in the future.

3.6.1.
"What would make the Data Hub the most useful source of Sierra Leonean education data, as far as you're concerned?" The consultants and university academics we interviewed were quick to point out that the ability to download data would be their most preferred immediate improvement.
The next most frequent request was for more data with more granularity. Gender, dropout, and teacher data were raised several times, along with broader employment, health and societal data, in particular, to engage more with civil society by providing strong evidence of the efficacy of getting an education.
"In developing countries, education is an economic means, does this education get me a job?"

Insight:
There is a huge appetite among users for validated, official data. Some users could see the Data Hub becoming a future source of all data and information on education, health, population trends, employment, finances, and more.

Strategic questions:
• Where does the remit of the Data Hub end?
• How would it hand over the transition to other data sources (for example to the Ministry of Labour and Social Security, the Ministry of Health and Sanitation and other governmental agencies)?
• How would it avoid being 'taken over' by data from other entities over which it has little control?
• Could -and should -the Data Hub be set up as a service that could be replicated for other policy areas outside of Education?
Users also shared the view that it would be helpful to have a way to ask questions, flag data inconsistencies (for example, a participant searched for their school but the spelling was wrong on the Data Hub and they had to find it through filtering) and missing data, or to signal performance issues (for example, a participant asked why the Data Hub is often down in the middle of the night.
Many users shared that they currently use back channels to find someone to ask about data or the Data Hub, and sometimes failed to find the support they needed.
Some users outside of Sierra Leone also shared that they didn't know who the DSTI were and were unfamiliar with the relationship between the DSTI and MBSSE.
Lastly, current users of the Data Hub indicated that they would like to be informed of data updates and any other changes to the Hub.

"What could be the worst thing the Data Hub could evolve into, as far as you're concerned?"
Our interviewees were honest in reflecting on what they feared could happen to the Data Hub, based on previous experiences with similar endeavours. A sample of responses to this question is highlighted below.  Only one NGO / IGO representative indicated using a mobile device because they did not have access to a computer connected to the internet to look at the data that would help them with programme decisions. All other interviewees could use a computer with internet access. However, the single mobile phone user that we encountered is likely to be representative of many more. Further research is required to better understand the on-the-ground needs of local NGO and IGO representatives.
From the user interviews, including one with a parent, a case can be made for making at least a section of the Data Hub specifically parent-facing and mobile-friendly, for example by building a 'Find a school' mobile-first service. This would complement the existing USSD services already available to parents.
We do not recommend making the entire site mobile-responsive, and accessible via mobile phones, as it was not a stated recommendation from most users -NGOs, IGOs and academics -who prefer to download and analyse data using a computer. Making the entire site responsive could threaten the existence of sophisticated graphs and filters which they value. We recommend creating a stand-alone, sister mobile service which could help drive usage from parents and other users in Sierra Leone who may be interested in specific school performance and characteristics, although this is not an obvious need as parents are able to access this type of information through other informal sources.
Going forward, a mobile-first service could include: • Mobile-optimised: small screens, big fingers; • Low page weight including low-fidelity school imagery: usage doesn't eat up the users' data allowance, and service that runs well over low bandwidth; • Copy that is written for low literacy and low numeracy (the exact target level would need additional definition); • No graphs (since graphs often do not display well on small screens, and could be communicated through words instead); • Quick and simple navigation to a school: few top filters, secondary advanced search; • Selected data sets with simple explanations of how they show what is going on (dual purpose of education on the situation and on how to read a graph); • Additionally, it could help to answer use cases such as: • This could probably be a web-app for lower costs, to be defined with an engineering team; • The value proposition to be tested could be 'As a parent or as a former student, I want school data at the tip of my fingers to check my school and show it off'; • Currently, parents use school open days and word of mouth to select a school, then get performance information about the school their child(ren) enrolled in from the CAR (Continuous Assessment Recordkeeper) teacher; ○ Consider linking the Data Hub to official information on enrolment (see for example how Australia does it .) 13 The existing desktop version would supplement the mobile content, with a richer data set and complex functionalities for the benefit of researchers, NGOs and consultants who have fewer or no internet data and device size constraints. The data sets accessed on a desktop should contain: • High fidelity school imagery (this was highly rated by many participants); • Graphs; • Ability to filter to groups of schools; • Ability to download parts of the data, not just all or only school by school.

Figure 5. Insight and strategic questions 5
Insight: To serve parents and Sierra Leone based users with simple queries (How's this school doing? What's the best school in that area?), the Data Hub needs to deliver some of its content in a mobile-first manner. To serve its other audiences, it needs to deliver its sophisticated features and new ones, such as downloads, in a desktop-first format

Strategic questions:
• Is it reasonable to consider delivering part of the data via a dedicated mobile app or website centred around school and geographical area data, while maintaining all of the data accessible on a desktop-first version in order to continue providing the much appreciated 'at a glance' graphs, in particular.
• What are the edge cases at the boundary between mobile and desktop usage?
• Are computer and internet penetration likely to drastically change over the next year, making the case for a mobile-first version less relevant?

Beyond the user research findings
We were provided with data from Cloudfront, which is the content delivery network for the Data Hub, for web traffic to the Data Hub between 5 April and 3 June 2020.
This data shows content requests by visitors to the site and comes from server requests, rather than embedded trackers on the front-end. This is therefore not an extensive set of analytics data captured on the pages themselves. For example, ideally, analytics data shows number of clicks, page dwell times, bounce rates , and journeys 14 across pages.
What this data reliably tells us is that 90% of users of the Data Hub in that period are located outside of Sierra Leone, the vast majority in the USA.
After excluding the 30% of traffic from bots, remaining traffic is as below. Please note that these are not unique visits.

Figure 6. Distribution of visits to the Data Hub between 5 April and 3 June 2020
58% of non-bot visits are from desktops 30% of non-bot visits are from mobiles Remaining visits are from tablets and unidentified platforms.

What we expected to uncover and didn't find
At the outset of this research activity, we identified a series of beliefs and starting hypotheses in partnership with the MBSSE and DSTI that gave initial direction to the research. We agreed that we would not focus on validating these hypotheses specifically, but rather use them as a starting point for our discussions and refine our understanding based on the insights we gather.
Through these discussions we learned that two of our starting hypotheses were not validated.

Use case A: Policymaking and policy efficacy assessment. Usage of the Data
Hub by policy assessors is hampered by the fact that it displays 2018 data. We expect this use case to become valid once the Data Hub is more up-to-date.

Use case B: School enrollment decision making.
One respondent mentioned knowing "some parents who looked at the Data Hub to choose a school" but we were not able to interview those parents directly. Furthermore, between the challenges associated with accessing the Data Hub via a mobile device, its 2018 data, and a firmly established process of parents sending children to the schools they attended, and finding out about quality schools through word of mouth, we believe the Data Hub isn't yet a viable primary source of enrollment decision data for most parents across the country. We have anecdotal evidence that civil servants use internal or back channels to access education data, and feel the Data Hub is not targeted to them.

Strategic questions:
• Is one of the aims of the Data Hub to serve the data needs of civil servants, replacing existing internal channels and becoming the 'single source of truth'? • Is it realistic to expect this change would be welcome and embraced by requesters and users of education data in civil service?
Further to these starting hypotheses, as the research progressed, we expected to hear more about how the data could be used for advocacy purposes and have not found this to be the case among our participants. Current usage is mainly for programme planning and funding proposals.
Based on our initial conversations with stakeholders at the MBSSE and DSTI, we also expected to learn that school staff (principals, board members and teachers) were using the Data Hub to check the status of their school and surrounding schools. The inaccessibility of the Data Hub on a mobile device seems to be one of the main reasons for its low to non-existent usage among this group of target users. Further study is required to better understand the needs of these groups of users.
Finally, we did not uncover or validate any access needs.

Recommended next steps
This section sets out four types of recommendations, roughly listed in priority order: 1. Immediate upgrades to the Data Hub: These are no-regret, near-term improvements that will help the Data Hub to better meet user needs.

Define the future of the Data Hub:
We would recommend the DSTI works with partners to define the mission and goals of the Data Hub, clarify the scope and ensure it has the team, skills and budget to support continuous improvement, with clear ownership.

Focus on expanding awareness of the Data Hub
and increasing traffic and usage: Once the most pressing improvements to the Data Hub have been made, and its future and has been secured, we recommend a focus on driving usage through clearer naming of the service, and optimising it for search engines.

Develop a roadmap to support future delivery:
To help plan for the medium to long term, we have recommended some product management approaches.
Please note that while recommendations are primarily based on the user research findings outlined in this report, they also draw on the user research team's experience of delivering digital products and services and international good practice.

Recommendation 1: Immediate upgrades to the Data Hub
We recommend that the DSTI makes the following improvements to the Data Hub in the near term. The precise prioritisation and timelines for these upgrades should be decided by the DSTI and BWS, informed by the budget, team and time available, as well as policy priorities.

1.1.
Add information about the DSTI, the source of the data displayed and the analysis behind data and graphs. An 'About us' page and relevant links from and to other governmental websites would help to build authenticity and authority

Other product changes to consider
We also identified three product changes to consider in the longer term. These are less urgent and require more exploration, than those listed above.
• Explore how to satisfy, if at all, the respondent's request for more data and more granularity in the data.
○ The Data Hub could also allow the upload of data as a way to increase its value and role to all users, particularly academics, NGOs and consultancies that regularly produce their own data sets and reports and aim for wide distribution and usage. We recommend applying Postel's Law: Be strict in what you put out and be lenient in what you receive. For example, the difference in authorship would need to be both clear and obvious.
• Consider how to visually present data for the various audiences of the Data Hub ○ How can the needs and abilities of an audience comfortable with data (such as analysts and academics) be balanced with the needs of non-specialist audiences?
○ How can the key data points that are provided as 'ready baked' graphs make their importance and value clearer to users?
○ How customised or customisable should graphs be?
○ Can graphs be downloaded? As a data set, as an image?
• Consider delivering part of the data via a dedicated mobile-first website: for example, school and geographical area data.
○ What are the edge cases at the boundary between mobile and desktop usage? Are computer and internet penetration likely to drastically change over the next year, making the case for a mobile-first version less relevant? ○ Example mission statement:

Recommendation 2: Define the future of the Data Hub
The Education Data Hub collates, hosts and distributes digitally Sierra Leone-centric education data in raw and analysed formats, to enable all users to efficiently retrieve official data and information.
○ Example set of goals: ■ To be the first and unique point of call for all Sierra Leone education data requests ■ To be the most reliable, efficient and effective online source of official Sierra Leone education data ■ To be a recognised leader in education data distribution initiatives in government 2.3 Work with other parts of Government to clarify the edges of the Data Hub.
Platforms like the Data Hub are markedly more modern-looking and easy-to-use than other websites from the same organisation (or parent organisation). A classic pitfall for such platforms is that they are viewed as excellent digital packhorses for all 'sort of the same' content from various other entities, content that is actually unrelated to the mission and purpose of the host site and also not budgeted for.
○ The stronger and clearer the strategic purpose of the Data Hub will be, the easier it will be to not stray too far from the intended scope and preserve and deliver the Data Hub's original value proposition. ○ The product owner should have access to regular time from a full-stack developer and a designer with user research skills, at minimum, in order to be able to independently roll out fixes, improvements and updates.
■ Ideally, a multidisciplinary digital team within the DSTI would maintain and continuously improve the Data Hub as part of a portfolio of digital products and services.

Recommendation 3: Focus on expanding awareness of the Data Hub and increasing traffic and usage
Higher awareness, traffic and usage is likely to support the case for future funding. Therefore, consider focusing on these areas before investing more time in planning and making product improvements.

3.1
Consider renaming the Data Hub, to make its purpose clearer to users, and making a distinction between the service provided and the platform providing it.
○ As a service, the Data Hub enables free access to nationwide education data. As a platform, it hosts and distributes that data.

Notes
Here are some more ideas based on what we have seen work elsewhere. The DSTI will need to experiment and test what is most effective in this particular context.
• Ask trusted government websites and digital services to link to the Data Hub, where relevant. • Include the URL on communications to target user groups, e.g., in SMS messages or letters sent to parents and headteachers. • Consider blogging about product improvements and user research to signal transparency, improve visibility and engage your most active users. • Trial an email newsletter summarising major updates or upgrades.
• Encourage third-party advocacy by providing social-media-friendly snapshots of data that are easy to share to raise awareness, as Ziauddin Yousafzai, father of Malala, does for Pakistan's schools: • Note that Mr Yousafzai received multiple inquiries about the source of such data in his Twitter feed. We recommend creating a snapshot template that clearly links back to the hub with a logo and a (possibly shortened) URL.

Recommendation 4: Develop a roadmap to support future delivery
Our fourth set of recommendations concern the development of a roadmap to support future delivery. This recommendation has the following aspects:

4.1.
Determine what success (the vision) looks like for the Data Hub, and how the team will know it's on track for that success. 4.2. Define metrics that let the team know whether the service is performing within expected parameters (commonly known as KPIs, Key Performance Indicators). A typical KPI is website uptime, the time that a website or web service is available to the users over a given period. Knowing that some downtime is expected, most providers try to meet a goal of 99% uptime or higher. 4.3. Define objectives and key results (OKRs) that let the team know if they're on track for success and when they have succeeded.

Notes
A good OKR is directional: "By the end of the quarter, we'll have reached the summit of Mount Everest". For the Hub, an example OKR could be "By the end of 2020, we'll have made all available data downloadable in customisable sets". Build, then maintain, a product roadmap based on the vision, mission and objectives: • A product roadmap is a high-level, strategic document that maps out general stages of a product's development. The main purpose of a product roadmap is to tie a product's vision in with its parent organisation's objectives.
• A product roadmap is created as a result of strategic planning. It documents both the executive strategy and overall goals of a product. should outline how you intend to improve the Data Hub now, next and later (for some further reading on product roadmaps we recommend mindtheproduct ) 24 The most useful roadmaps are pragmatic and define outcomes and sequencing rather than milestones and dates. It should be regularly updated as time passes, assumptions are tested, and work is done. The product team should plan to conduct regular usability and accessibility testing to evaluate if the Data Hub is fit for purpose, and identify desirable updates and changes.
To gain the best insights into usage needs, habits and patterns, we conducted remote semi-structured interviews, using a set of uniform questions that were asked of all interviewees. The interviews were supplemented with additional questions as needed for further clarification or to request more details.
As research went on, we learned that many interviewees had little to no experience using the Hub, and in some instances no knowledge of it. Our research questions were modified accordingly, and not all questions were asked of everyone.

Identifying research participants from available traffic data
We looked at data traffic reports for the period 5 April-3 June 2020 to glean insights about the types of participants who would be best positioned to engage with us on the user research.
These reports give some information about traffic through the CloudFront server but contain no information about the activity on the website itself. An analogy would be that server traffic counts how often a door into a room is opened but does not say who it was or what precisely was done in that room. In the digital world this information would be provided by client-side trackers (cookies), elements in the page code that provide information on everything done by a visitor on a webpage, and tell us whether visits are by different or unique users.
Through cross-referencing the enquiries that the DSTI receives and the traffic data, we were able to build an initial picture of who was visiting the site but with few actionable details and insights. We learned that the traffic was largely coming from outside of Sierra Leone (10% Sierra Leone, 90% abroad) so we decided to overweight potential national users in our sample, in order to better understand their needs and use cases.
The server analytics document a total of 277 visits from external referrers for a 60-day sample period, which includes Google, Facebook, URL shortener t.co and other sources referencing the hub such as otrasvoceseneducation.org and worldscientific.com , 25 26 however gave no information on whether multiple visits were from the same machine ('unique visits').

Number of research sessions conducted and representativeness
The initial scope of work for the user research was based on 15 interviews, as a snapshot of insights. This goal was to provide early insights in time for the next release of the Hub rather than focusing on identifying a representative sample of users.
In the absence of unique visitor numbers, we decided to randomly sample individuals or teams from the following groups: • known visitors, who had been in contact with either the DSTI or MBSSE about the Hub; • likely visitors, who might need to access the data on the Hub because of their specific roles or functions within the civil service and IGOs, and some NGOs; • 'could-be' visitors among civil society who might need the data on the Hub for their work or family.
We increased our sample size to 19 interviews to improve the representativeness of our respondents, a need we identified early in the research process, and to accommodate the strong response to our outreach efforts.
However, with the current lack of precise visitor traffic, we cannot evaluate if our sample is representative of the overall population of visitors.

Details of the sessions conducted
We conducted 19 unique individual and team interviews between 11 and 24 June 2020. The interviews were conducted remotely using Zoom and WhatsApp. On average, the interviews took approximately 40 minutes.
As recruitment is a common challenge in user research, we are grateful for the assistance from the DSTI and the EdTech Hub in identifying suitable participants (i.e., diverse representatives from the user groups mentioned below).
All interviews were led by senior user researcher Sophie Freiermuth and by principal consultant Claire Bedoui from Public Digital.
Project stakeholders Abeba Taddese from the EdTech Hub and PJ Cole from DSTI attended one interview each.

User groups
Prior to beginning our interviews and based on initial research and conversations with the MBSSE and DSTI team members, we assumed the Data Hub had three main groups of users -civil society, civil servants, and NGOs and IGOs -that were likely not evenly representative. Following a discussion with the project steering group halfway through recruitment efforts, we decided to prioritise users from civil society over civil servants. The MBSSE stakeholders were keen to know how the Data Hub could be useful to Sierra Leone's citizens, as 'end-users' of education policies and reforms.
The primary purpose of the three-part classification mentioned above was the recruitment of diverse current Data Hub users. It is not intended to establish or imply fixed user categories and is expected to change as the use of the Data Hub and frequency evolve over time.

Civil society
The civil society group is composed of: • Parents of children of or nearing school age • Private organisations in the education sector such as ○ consultancies who work either with NGOs, intergovernmental organisations or government agencies, ○ businesses that provide products and solutions rather than delivering education services. • Academics • School principals / headmasters and mistresses

Civil servants
The civil servants group includes individuals in charge of assessing the quality of the education delivered (e.g., school inspectorate representatives), as well as those in charge of school infrastructure (buildings and facilities), and those in charge of global education policies and their evaluation.
This group also includes two respondents who work as independent consultants, primarily or exclusively for the government, as their data needs are directly related to their job in government.

Non-governmental and intergovernmental organisations
In this group, we clustered organisations of all sizes, from a large multilateral organisation (UNICEF) to a medium-size organisation with offices in developed countries, to a small and new Sierra Leonean NGO.

Recruitment: targets and final tally
The tables below show our initial main categories and final sub-categories. To complete our 19 interviews, we conducted outreach to 28 individuals and organisations. Among the interviewees, 13 were based in Sierra Leone. The six others were based in the United States, the United Kingdom and Jordan.

Notes from a usability test
During one interview, a participant went to the Data Hub and shared their screen with us on the call. We observed them freely using the Schools Directory and conducted an ad-hoc, unscripted usability test. Review the position of the button and its label in relation to the rest of the content Search failed as their desired school has a typo in the data Search engine should offer 'did you mean' as well as predictive search The search bar was their preferred way of exploring the data, despite not providing them with the desired results If search engine cannot be updated and filters are the best way to search, consider making the search bar less prominent Struggles repeatedly to understand that once a filter has been selected, a second filter displays and needs to be activated Review the UI of filtering. Consider replacing dropdowns by radio buttons or checkboxes. Some of the filter options are very long, might need to be reviewed Eventually filters down to their School.
They spend a lot of time looking at the photos Consider allowing a lightbox or zoom functionality to showcase the photos.

Other issues raised by interviewees
Parents expect the MBSSE to provide information on the grading system, "what do [sic] a student need to score before he / she's awarded an A or Excellent?" Government-owned schools, from primary to mid-secondary have been devolved to the Freetown city council. They are supposed to have supervisory responsibilities but there are real challenges with implementation of the devolution. For example, the decision to close schools was taken without engaging with the city council. 'Leh wi Lan' wasn't communicated to the council. Until the mandate to the city council is clarified, and the relationship between city council and schools is unequivocally defined, the council has little use for the Data Hub.
The school principals and civil servants we interviewed mentioned their difficulty in attracting and retaining qualified, competent and engaged teachers. They expected the Data Hub to include data that would help them.

Lessons for future user research
We strongly recommend this exercise is repeated when the 2019 data is uploaded and hope this report will be used as a baseline for comparison purposes.
It was easy to interview users who were not based in Sierra Leone as internet connections were reliable. In contrast, the interviews we conducted with users in Sierra Leone were challenged by low bandwidth, which on occasion required us to switch from using Zoom to WhatsApp. Unlike the Zoom calls, the calls on WhatsApp were not recorded, which made excellent note-taking critical.
We recommend conducting continued contextual enquiries and ethnographic observations of target users in their activity with data, in order to better refine the interface and content. Some of these studies should be longitudinal.
We also recommend further continuous usability and accessibility testing that is preferably conducted in-person, in order to inform design decisions that will deliver a version that is accessible to 'the man on the street', as poetically described by one of our interviewees.