Hacking the knowledge of maker communities in support of 21st century education

. The paper addresses the need to rethink education to be eﬀective in a changing environment. More concretely we look at the intersection of craft-based learning, digital fabrication technologies and schools’ capacities to absorb educational innovations. Although making and hacking are known activities within constructionist learning settings, they are not yet widespread at a school level. An explorative study of maker education across European countries has shown that a major impediment to innovations, such as digital fabrication in schools, were the perceived complexity of the process, the technical skills required and the lack of easily accessible resources for getting started or being able to troubleshoot if needed. The aim of this paper is to test the possibilities of referencing existing knowledge embedded in platforms such as instructables.com. Using the available API, we created a network graph of 225,681 instructables authored by 74,824 authors. The potential of that knowledge base is analysed in two steps: ﬁrst, we describe the available content on the platform in terms of topics, structure and licenses and second, we explore the value of topic networks, as one speciﬁc possibility to make platform knowledge more accessible to educators and learners themselves. A ﬁrst prototype has been implemented and evaluated, showing the importance of discussing the value and limitations of resources external to educational systems, learning by doing, accountability and the right to tinker in technology-embedded teaching.


Introduction
Looking back in history, every society over the different centuries had its specific demands in terms of skills requirements at the workplace and as a result in terms of education. Thus, broader societal developments tend to influence pedagogy, in terms of teaching methods as well as in terms of educational goals associated with specific skills and competences, required to enter the workforce [14]. With ever more fluid demands on educational systems comes the challenge of adapting curricula, educational technologies and resources [29]. Our aim is to open up existing resources by testing the possibilities of using content from platforms such as instructables.com. More concretely, two questions will be addressed: (a) Given that emerging or relatively small communities might not yet have a seizable stock of resources related to technology enhanced learning topics, is it possible to use knowledge from more established communities as a starting point, being aware that communities such as instructables.com cultivate a much more informal learning dimension than what would be typical for schools. (b) Content on established maker platforms goes into the hundred thousand and more. Therefore, when using content from established maker platforms, can we establish a transparent and effective way to select content that fits users' needs and results in resources being perceived as useful to learners and teachers?
The paper starts with analysing the potential of Instructables.com's knowledge base in two steps: first, we describe the available content on the platform in terms of topics, structure and collaboration behaviour (chapter 4) and second, we explore the value of topic networks, as one specific possibility to make platform knowledge more accessible to educators and learners themselves (chapter 5). A first prototype is then described in chapter 6, followed by a brief evaluation section. The paper concludes by discussing the value and limitations of resources external to educational systems, learning by doing, accountability and the right to tinker in technology-embedded teaching.

The Demands of 21st Century Education
The 21st century is characterised by a shift of the workplace towards automation, globalisation and demographic change [17]. Computerisation and automation are seen as drivers for "routine" tasks and "lower skilled" jobs becoming obsolete in a near future. Areas where human workers are not yet easily replaced by automation include solving unpredictable problems and maintaining complex interactions with other humans [17,29]. Globalisation has led to a situation where services such as the ones offered by call centres are outsourced to countries with lower wages and where workers compete on global scale for jobs [12]. Spurring creativity and innovation is seen as key to compete in a globalised job market. Demographic change has an impact on the employment market in many ways: in terms of superannuation of the population, which will require more jobs in health and foster care; and in respect to diversity of the society [4,18].
Furthermore, most of the workplaces are characterised by less hierarchy and supervision and allow for more autonomy and responsibility and require working in teams. Workers are asked to continuously adapt to new challenges and demands and train themselves in new skills. Thus, one of the requirements in the current and near future employment market is the skill to know how to learn [7,13].

Maker Pedagogy
We can observe that the pedagogy in schools is already paying attention to these changed circumstances to a big extend. Learner centred teaching methods, constructive learning approaches, project based and collaborative learning have at least partly replaced ex-cathedra teaching, so that learners of the 21st century are encouraged to become creative, critical thinkers, skilled in communication and team work [3].
The so-called 21st century skills comprise a vast set of skills and competences complementing traditional skills. Frequently mentioned skills include [10]: critical thinking and problem solving; creativity and innovation; communication and collaboration; digital literacy consisting of information, media and ICT literacy; and finally interpersonal skills such as flexibility and adaptability, initiative and self-direction. One of the conclusions of the report by Jerald [17] is, that these "new" skills have become more important, but that they should complement rather than replace the skills pursued in traditional school subjects such as mathematics, grammar, and so forth. Hence, 'novel skills' must be trained not detached from the traditional school curriculum but as part of "traditional subjects". Many of the skills that are associated with 21st century skills are supported by the maker pedagogy, often integrated in project-based settings.

Tinkering versus Instruction
As put forward by Resnick and Rosenbaum [27]: "The tinkering approach is characterized by a playful, experimental, iterative style of engagement, in which makers are continually reassessing their goals, exploring new paths, and imagining new possibilities. Tinkering is undervalued (and even discouraged) in many educational settings today, but it is well aligned with the goals and spirit of the progressive-constructionist tradition-and, in our view, it is exactly what is needed to help young people prepare for life in today's society". The quote shows how existing preferences in the educational system, e.g. emphasizing content delivery and quantitative assessment, run counter to a pluralism of learning paths including the bottom up experiences of creating tangible objects, the notion of adapting solutions to changing conditions and an essentially different way of accessing STEM problems.
The pedagogy of making builds on pedagogical schools of thoughts ranging from reform pedagogues to constructivists, from Montessori [24] to Piaget and Papert [1], who emphasized the value of self-regulated learning [16], empowering learners to decide on their learning goals and the ways to achieve them. In such learning settings teachers see their own role primarily in assisting the learner in their learning paths. The focus on instructional interactions in traditional teacher-learner relationships is therefore obsolete.
Learning through making is hands-on learning, where makers learn from observing others, recognising the value of trial and error, and taking part of interdisciplinary and collaborative teams [6,19,9]. In this, making is similar to problem solving and project-based learning approaches. Making includes a desire to produce things more collaboratively by improving design suggestions of others or by simply copying, mashing or personalising existing design elements [31]. Making is thus theoretically and historically founded on "learning by doing" principles [25,26]. According to the Horizon report, which anticipates technological trends having an impact on educational settings, maker education will have an increasing impact on education in the following years [5].

Implications for Educational Systems
A recent study of innovations related to making and digital fabrication in schools [32] identified a number of barriers to innovating teaching practices, including a lack of knowledge about how to run the technology or how to integrate the use of maker technologies with curricular topics to be covered. Those barriers, in combination with cumbersome decision-making processes and restrictive funding options, could seriously hamper the use of novel technologies in schools. In order to overcome these limitations, educators mentioned several measures such as the use of sharing platforms or groups of like-minded people sharing their experience. From an innovation management perspective, technologies that enabled networking between innovative teachers were largely preferred over knowledgebanking strategies such as best practice collections or 'go-to experts' [32].
In itself, referring to networking and community building as promotional devices of educational purposes is not new. Innovation networks in education can be seen as an amalgamate of Wenger's communities of practice [34] and Kazmer's idea that knowledge is shaped by learners' membership in multiple overlapping communities [21]. The benefit that comes with considering communities is primarily based on their purpose giving nature, even though we must be aware of overstating the homogeneity of communities [33]. Wenger argues that throughout life, communities motivate our learning by defining the relevancy of problems and providing orientation in terms of where answers can be found or where previous attempts to find solutions have been unsuccessful. On a more practical level, however, we do not yet fully understand what makes collaboration in communities work [28]. Kaptelinin [20] argues that collective activities are structured, directed and motivated by objects, which capture the purpose of networking . Hence, objects in networks help individuals to express themselves in a cognitive as well as affective way -a process Knorr Cetina refers to as "object-centred sociality" [22]. Taking project descriptions as objects' enabling the shaping of knowledge and communities, the following section describes activities related to those objects.

Platform Activities on Instructables.com
Instructables.com is a platform enabling users to upload and share their do-ityourself projects, which are commented upon and rated by other users for their quality. The website was launched in August 2005 and obtained by Autodesk in 2011. Figure 1 shows a relatively moderate start during the first 5 years, but since then monthly uploads have been overall on the rise. Instructables.com promotes a specific format for writing instructables -e.g. project descriptions -, including a step-by-step description and images for in-between results. Data on authors and their projects can be accessed via an Application Programming Interface (API). By October 2017, users had published roughly 250,000 projects, from which we obtained a dataset of 225,681 projects, published and shared by 74,824 distinct first-authors. If we look at the distribution of published projects per platform user we see a long-tail distribution, fitting best a log-normal distribution (Figure 2). Figure 2 shows two density functions, with the continuous line indicating the actual distribution of projects per user and the dotted line shows a lognormal distribution. The long-tailed distribution has been tested as described in [8] using the power law python package [2]. A practical interpretation of a long-tailed distributions is to see it as an indicator for a slow or maybe not too intuitive start, representing the large majority of users having just one project published. Figure 2 shows that only a few active users, namely the top 20 users by the time of analysis, had more than 200 projects. The most prolific user had about 500 projects, mostly in the crafts, gardening and food categories. However, projects tend to be mixed and in another case of a user with 179 projects, we can see a variety of themes including minimalist LED clocks, photography and chickpea meals. Furthermore, screening the top 20 users, it seemed that they had professional incentives to produce instructables, either as teachers, small businesses, makers or managers of makerspaces.
Early on we discarded the idea of selecting authors based on their project profiles, this would go against our intention to identify technology projects embedded in other areas such as gardening or art. We also looked into the selfdescriptions of users in order to see whether there was a discernible education community. Without claiming that such an approach could not be promising, a first frequency count of keywords generated rather low numbers: student (1226), Lastly, platform activities can be described in terms of ongoing collaborations, i.e. visible by the connectedness of an author. Connectedness refers to the number of people, an author has collaborated with. The rationale behind this measurement is that authors who contributed to more projects are more experienced and that project descriptions that were generated in collaboration with others, most likely have had an internal peer review process increasing the quality of the descriptions.

Content: Types, Categories, Channels and Tags
Instructable.com content can be separated into two types: projects (97 %) and guides (3%). Guides contain themed collections of projects around topics such as micro-boards or magnetism. Each project is allocated to one out of eight categories, with technology, craft and workshop being the most popular categories (see Figure 4, left). Originally content focused mostly on projects such as building electronic or mechanical devices to solve common problems around the home. The scope of the project has then expanded to include less technical categories, including Food, Living, Outside or Play.
Over time, the category 'play' was used less and, as expected, the category 'costumes' shows seasonal dependencies. At a more detailed level projects are organized into channels and can be described with key-words. Arduino is both the most frequently used channel (see Figure 4, right) and the most frequently issued keyword, which again highlights the community's technical affinity.  So far, we analysed options to filter content on instructables.com by categories and channels, which is a useful first step to narrow it down but does not leave many additional options for finding projects beyond pre-established categories. For the next step we generated a network graph based on user-allocated tags ( Figure 5).
The size of each node reflects the frequency of the 'tag' and if two tags are connected then this means that at least on project uses both tags. This way, for example, it will be possible to find projects that use the tags 'solar', 'lego' and 'iPhone'-tags that could be relevant in a school context -leading to a project that describes building a USB charger in a casing made by Lego bricks. Another characteristic of project descriptions is 'step count', i.e. the number of steps used to describe the process of producing a specific outcome. Figure 6 shows a histogram of steps per project.
An interesting question would be whether 'step count' is positively related with pedagogical quality. 18% of all projects have only a single step, otherwise most project use five or six steps to break down instructions within project descriptions.

A Tag-based Recommender Prototype
Providing the tag-based content network to end-users is close to providing a first prototype of a recommender systems as discussed in [23]. Manouselis at al. distinguish different recommendation goals, including recommending 'content related to a learner's current interest' and 'content related to specific topics', both scenarios can be covered with a tag-based content network as described in the previous section.
The user interface of the prototype is shown in figure 7. Users can select tags either from a list or by directly clicking on a node. As shown at the bottom of figure 7, a list of projects, ranked by likes, is generated, depending on the tags selected. All dots are color-coded: orange means that a tag is already selected, grey means that there is no project using the already selected tags and yellow means that there are projects using this specific combination of tags.
An architecture sketch of the system behind the user interface is given in figure 8. However, we are aware that more complex recommendations such as recommending a preferred pathway through a list of resources would require observing users' interactions with the system and modelling users prior knowledge about a given topic.
At the moment our prototype can filter resources by topics but does not take into account yet the importance of selecting materials with an adequate level of complexity so that users are neither over-nor underchallenged [15].

First Evaluation Results
The first objectives to be achieved with the prototype were related to makers' or teachers' needs to find resources which addressed the functioning of specific technical components (e.g. LED, DC motor, micro-boards) or tools for designing and programming (e.g. tinkercad.com or create.arduino.cc) in an applied context (water, garden, green, reuse etc.). At this point, it is important to be aware that the tags are attached to concrete objects and projects, hence more abstract tags such as 'sustainability', 'geography' or 'equity' are less frequently used.  Based on Manouselis et al.'s [23] overview of recommender systems in technologically enhanced learning settings, we established the following evaluation criteria: degree to which a recommended source uncovered hidden aspects of a topic, degree to which a recommended source included technical as well as pedagogically useful ideas and explanations, likelihood that the recommended source would actually be used and likelihood that the recommended source would need considerable reworking before it could be used.
In order to support the data collection for these evaluation criteria, a workshop with 12 teachers and tutors was organized. During that workshop we discussed the need for diverse types of resources helping teachers not only with the technical aspects of digital fabrication technologies in classrooms but also with the logistical or pedagogical implications of using specific technologies with students. Part of the workshop was an exercise where participants were asked to imagine their preparation activities for a class involving digital fabrication and choose between two and three tags that would adequately describe their needs for resources. They were then asked to visit the first two projects recommended and rate them according to the statements provided in figure 9. Number one to five on the provided Likert scale equal 'very much disagree', 'disagree', 'neutral', 'agree' and 'very much agree'. In two cases, the selected tags produced a list with a single resource only, hence we got feedback for 22 recommended projects. As shown in figure 9, overall the recommended resources were perceived as useful enough to be integrated in a classroom. The possibility to extract pedagogical knowledge from projects on instructable.com was rated the least favourable, with a majority of respondents disagreeing with the statement 'The resource included helpful pedagogical knowledge'. However, that was to be expected since users of instructables.com see themselves first and foremost as technical experts or tinkerers and less as educators. Nonetheless, the pedagogical value of stepwise explanations, including the necessary details to also replicate a project step is a frequent topic in comments. With 50% of the resources, workshop participants agreed with 'I discovered opportunities I was not aware of before. ' We also see this as one of the main benefits of looking into existing project descriptions, the chances that a project matches a specific user problem are rather small but providing inspiration to a teacher in search for a meaningful example or application area is a reasonable expectation. One common question was whether querying the tag-based network and searching through instructable.com's search engine would provide the same results. Indeed, finding resources is the main business of search engines. However, preliminary tests indicate that whereas instructable.com works fine for frequently used combinations such as 'arduino' and 'water', leading to similar results of automatic plant watering systems, less frequently used combinations such as 'arduino' and 'jewellery' are better served by relying on 'tags' provided by users.

Conclusion
At this stage it is still early for a final verdict on the overall value of hacking community knowledge to empower educational communities on their path to more hands-on learning, including digital fabrication tools. The current evaluation happened in a workshop, however, the final recommendation tool is planned to be integrated into a larger support system together with links to open educational resources, tool recommendations for learners and the possibility to support learning analytics running in the background. In such a context, feedback can be given online and larger numbers of users are addressable.
Revisiting the research questions posed in the first section, we would conclude that tapping into resources generated by relevant communities such as instructables.com is a valuable endeavour. The inspiration these resources can provide for teachers and learners as well as the technical details included in project descriptions was mentioned positively during the evaluation workshop described in the previous section.
Concerning the second question about a transparent and effective way to filter external resources for their use in an educational context, we would say that there are some promising options, but most likely direct feedback given by users integrating resources into their teaching and learning practices is needed to produce more accurate recommendations. For the moment, the list of recommended resources is ranked by likes. However, in the future we can also provide the option to rank recourses by the connectedness of their authors (cf. section 4) or the number of steps featured in a project (cf. section 5), or a combination of multiple measurements.
To conclude, repurposing the effort from the instructable.com community to benefit education is an effective way to avoid reinventing the wheel. What is left to do, however, is opening up platforms and proactively supporting the reuse of materials [30]. This is not limited to choosing creative commons licenses, which is already encouraged, but also includes supporting rich APIs and giving sufficient visibility to the information contained within project descriptions (e.g. project collaborations and descriptive tags are currently not visible on published instructables).
Last but not least, education and technology are not necessarily an easy match [11]. Teachers, who are traditionally seen as domain experts need to embrace a diversity of knowledge sources, skills acquisition and learning by doing are equally valid approaches and education systems need to encourage individuals to take on more responsibility for their own learning. These are changes that happen outside technical systems, however, are critical conditions for technical innovations to unfold as envisioned.