Journal article Open Access

Does Reviewer Recommendation Help Developers?

Vladimir Kovalenko; Nava Tintarev; Evgeny Pasynkov; Christian Bird; Alberto Bacchelli


DataCite XML Export

<?xml version='1.0' encoding='utf-8'?>
<resource xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://datacite.org/schema/kernel-4" xsi:schemaLocation="http://datacite.org/schema/kernel-4 http://schema.datacite.org/meta/kernel-4.1/metadata.xsd">
  <identifier identifierType="DOI">10.5281/zenodo.1404814</identifier>
  <creators>
    <creator>
      <creatorName>Vladimir Kovalenko</creatorName>
      <nameIdentifier nameIdentifierScheme="ORCID" schemeURI="http://orcid.org/">0000-0001-5880-7323</nameIdentifier>
      <affiliation>Delft University of Technology</affiliation>
    </creator>
    <creator>
      <creatorName>Nava Tintarev</creatorName>
      <affiliation>Delft University of Technology</affiliation>
    </creator>
    <creator>
      <creatorName>Evgeny Pasynkov</creatorName>
      <affiliation>JetBrains GmbH</affiliation>
    </creator>
    <creator>
      <creatorName>Christian Bird</creatorName>
      <affiliation>Microsoft Research</affiliation>
    </creator>
    <creator>
      <creatorName>Alberto Bacchelli</creatorName>
      <nameIdentifier nameIdentifierScheme="ORCID" schemeURI="http://orcid.org/">0000-0003-0193-6823</nameIdentifier>
      <affiliation>University of Zurich</affiliation>
    </creator>
  </creators>
  <titles>
    <title>Does Reviewer Recommendation Help Developers?</title>
  </titles>
  <publisher>Zenodo</publisher>
  <publicationYear>2018</publicationYear>
  <subjects>
    <subject>code review</subject>
    <subject>recommender systems</subject>
    <subject>user-centric evaluation</subject>
    <subject>empirical software engineering</subject>
  </subjects>
  <dates>
    <date dateType="Issued">2018-08-28</date>
  </dates>
  <resourceType resourceTypeGeneral="Text">Journal article</resourceType>
  <alternateIdentifiers>
    <alternateIdentifier alternateIdentifierType="url">https://zenodo.org/record/1404814</alternateIdentifier>
  </alternateIdentifiers>
  <relatedIdentifiers>
    <relatedIdentifier relatedIdentifierType="DOI" relationType="IsSupplementedBy">10.5281/zenodo.1404755</relatedIdentifier>
    <relatedIdentifier relatedIdentifierType="DOI" relationType="IsVersionOf">10.5281/zenodo.1404813</relatedIdentifier>
  </relatedIdentifiers>
  <rightsList>
    <rights rightsURI="info:eu-repo/semantics/openAccess">Open Access</rights>
  </rightsList>
  <descriptions>
    <description descriptionType="Abstract">&lt;p&gt;Selecting reviewers for code changes is a critical step for an efficient code review process. Recent studies propose automated reviewer recommendation algorithms to support developers in this task. However, the evaluation of recommendation algorithms, when done apart from their target systems and users (i.e., code review tools and change authors), leaves out important aspects: perception of recommendations, influence of recommendations on human choices, and their effect on user experience.&lt;/p&gt;

&lt;p&gt;This study is the first to evaluate a reviewer recommender in vivo. We compare historical reviewers and recommendations for over 21,000 code reviews performed with a deployed recommender in a company environment and set out to measure the influence of recommendations on users&amp;#39; choices, along with other performance metrics.&lt;/p&gt;

&lt;p&gt;Having found no evidence of influence, we turn to the users of the recommender. Through interviews and a survey we find that, though perceived as relevant, reviewer recommendations rarely provide additional value for the respondents. We confirm this finding with a larger study at another company. The confirmation of this finding brings up a case for more user-centric approaches to designing and evaluating the recommenders.&lt;/p&gt;

&lt;p&gt;Finally, we investigate information needs of developers during reviewer selection and discuss promising directions for the next generation of reviewer recommendation tools.&lt;/p&gt;

&lt;p&gt;Preprint: https://doi.org/10.5281/zenodo.1404814&lt;/p&gt;</description>
  </descriptions>
  <fundingReferences>
    <fundingReference>
      <funderName>Schweizerischer Nationalfonds zur Förderung der Wissenschaftlichen Forschung</funderName>
      <funderIdentifier funderIdentifierType="Crossref Funder ID">10.13039/501100001711</funderIdentifier>
      <awardNumber awardURI="info:eu-repo/grantAgreement/SNSF/Careers/PP00P2_170529/">PP00P2_170529</awardNumber>
      <awardTitle>Data-driven Contemporary Code Review</awardTitle>
    </fundingReference>
    <fundingReference>
      <funderName>Nederlandse Organisatie voor Wetenschappelijk Onderzoek</funderName>
      <funderIdentifier funderIdentifierType="Crossref Funder ID">10.13039/501100003246</funderIdentifier>
      <awardNumber awardURI="info:eu-repo/grantAgreement/NWO//2300187620/">2300187620</awardNumber>
      <awardTitle>Persistent Code Reviewing</awardTitle>
    </fundingReference>
  </fundingReferences>
</resource>
510
314
views
downloads
All versions This version
Views 510510
Downloads 314314
Data volume 1.8 GB1.8 GB
Unique views 455455
Unique downloads 284284

Share

Cite as