Info: Zenodo’s user support line is staffed on regular business days between Dec 23 and Jan 5. Response times may be slightly longer than normal.

Published October 23, 2020 | Version v1
Journal article Open

Crowdsourced Software Testing: A Timely Opportunity

Creators

  • 1. UST-Global, Inc.

Description

The concept of crowdsourcing has gained a lot of attention lately. Many companies are making use of this concept for value creation, as well as the performance of varied tasks. Despite its wide application, little is known about crowdsourcing, especially when it comes to crowdsourced software testing. This paper explores the crowdsourced software testing concept from a wider perspective ranging from a cost-benefit analysis, crowdsourcing intermediaries, and the level of expertise in the crowd. Drawing from a varied range of sources, a systematic literature review is done, where the research narrows down to ten most relevant peer-reviewed sources of high impact rating. In a comparative analysis between crowdsourced software testing and in-house testing, it is found that crowd testing has numerous advantages when it comes to efficiency, user heterogeneity, and cost-effectiveness.  The study indicates that intermediaries play a key role in managing the connection between the crowd and crowdsourcing companies despite various challenges. A comparison between novice testers and expert testers reveals that both the two have their unique capabilities in their respective domains.

Files

15.2.pdf

Files (413.1 kB)

Name Size Download all
md5:257b1702f72ad510fbffc0e7043f4410
413.1 kB Preview Download

Additional details

References

  • Billington, C., & Davidson, R. (2013). Leveraging open innovation using intermediary networks. Production and Operations Management, 22(6), 1464-1477.
  • Daniel, F., Kucherbaev, P., Cappiello, C., Benatallah, B., & Allahbakhsh, M. (2018). Quality control in crowdsourcing: A survey of quality attributes, assessment techniques, and assurance actions. ACM Computing Surveys (CSUR), 51(1), 1-40.
  • Donepudi, P. K. (2017). Machine Learning and Artificial Intelligence in Banking. Engineering International, 5(2), 83-86. https://doi.org/10.18034/ei.v5i2.490
  • Foong, E., Gergle, D., & Gerber, E. M. (2017). Novice and Expert Sensemaking of Crowdsourced Design Feedback. Proceedings of the ACM on Human-Computer Interaction, 1(CSCW), 1-18.
  • Leicht, N., Blohm, I., & Leimeister, J. M. (2017). Leveraging the power of the crowd for software testing. IEEE Software, 34(2), 62-69.
  • Leicht, N., Knop, N., Blohm, I., Müller-Bloch, C., & Leimeister, J. M. (2016). When is crowdsourcing advantageous? The case of crowdsourced software testing.
  • Leicht, N., Rhyn, M., & Hansbauer, G. (2016). Can laymen outperform experts? The effects of user expertise and task design in crowdsourced Software Testing.
  • Liu, D., Bias, R. G., Lease, M., & Kuipers, R. (2012). Crowdsourcing for usability testing. Proceedings of the American Society for Information Science and Technology, 49(1), 1- 10.
  • Muhdi L, Boutellier R (2011) the crowdsourcing process: an intermediary mediated idea generation approach in the early phase of innovation. Int J Entrepren Innov Manag 14(4):315–332. doi:10.1504/IJEIM.2011.043052
  • Stol, K. J., & Fitzgerald, B. (2014, May). Two's a company; three's a crowd: a case study of crowd sourcing software development, in Proceedings of the 36th International Conference on Software Engineering (pp. 187-198).
  • Zogaj, S., Bretschneider, U., & Leimeister, J. M. (2014). Managing crowdsourced software testing: a case study based insight on the challenges of a crowdsourcing intermediary. Journal of Business Economics, 84(3), 375-405.