Journal article Open Access

Does Reviewer Recommendation Help Developers?

Vladimir Kovalenko; Nava Tintarev; Evgeny Pasynkov; Christian Bird; Alberto Bacchelli


MARC21 XML Export

<?xml version='1.0' encoding='UTF-8'?>
<record xmlns="http://www.loc.gov/MARC21/slim">
  <leader>00000nam##2200000uu#4500</leader>
  <datafield tag="653" ind1=" " ind2=" ">
    <subfield code="a">code review</subfield>
  </datafield>
  <datafield tag="653" ind1=" " ind2=" ">
    <subfield code="a">recommender systems</subfield>
  </datafield>
  <datafield tag="653" ind1=" " ind2=" ">
    <subfield code="a">user-centric evaluation</subfield>
  </datafield>
  <datafield tag="653" ind1=" " ind2=" ">
    <subfield code="a">empirical software engineering</subfield>
  </datafield>
  <controlfield tag="005">20190409141049.0</controlfield>
  <controlfield tag="001">1404814</controlfield>
  <datafield tag="700" ind1=" " ind2=" ">
    <subfield code="u">Delft University of Technology</subfield>
    <subfield code="a">Nava Tintarev</subfield>
  </datafield>
  <datafield tag="700" ind1=" " ind2=" ">
    <subfield code="u">JetBrains GmbH</subfield>
    <subfield code="a">Evgeny Pasynkov</subfield>
  </datafield>
  <datafield tag="700" ind1=" " ind2=" ">
    <subfield code="u">Microsoft Research</subfield>
    <subfield code="a">Christian Bird</subfield>
  </datafield>
  <datafield tag="700" ind1=" " ind2=" ">
    <subfield code="u">University of Zurich</subfield>
    <subfield code="0">(orcid)0000-0003-0193-6823</subfield>
    <subfield code="a">Alberto Bacchelli</subfield>
  </datafield>
  <datafield tag="856" ind1="4" ind2=" ">
    <subfield code="s">5817258</subfield>
    <subfield code="z">md5:89d5916ce47d2cfc515ff87e8522c6cd</subfield>
    <subfield code="u">https://zenodo.org/record/1404814/files/revrec-preprint.pdf</subfield>
  </datafield>
  <datafield tag="542" ind1=" " ind2=" ">
    <subfield code="l">open</subfield>
  </datafield>
  <datafield tag="260" ind1=" " ind2=" ">
    <subfield code="c">2018-08-28</subfield>
  </datafield>
  <datafield tag="909" ind1="C" ind2="O">
    <subfield code="p">openaire</subfield>
    <subfield code="o">oai:zenodo.org:1404814</subfield>
  </datafield>
  <datafield tag="909" ind1="C" ind2="4">
    <subfield code="p">IEEE Transactions on Software Engineering</subfield>
  </datafield>
  <datafield tag="100" ind1=" " ind2=" ">
    <subfield code="u">Delft University of Technology</subfield>
    <subfield code="0">(orcid)0000-0001-5880-7323</subfield>
    <subfield code="a">Vladimir Kovalenko</subfield>
  </datafield>
  <datafield tag="245" ind1=" " ind2=" ">
    <subfield code="a">Does Reviewer Recommendation Help Developers?</subfield>
  </datafield>
  <datafield tag="536" ind1=" " ind2=" ">
    <subfield code="c">PP00P2_170529</subfield>
    <subfield code="a">Data-driven Contemporary Code Review</subfield>
  </datafield>
  <datafield tag="536" ind1=" " ind2=" ">
    <subfield code="c">2300187620</subfield>
    <subfield code="a">Persistent Code Reviewing</subfield>
  </datafield>
  <datafield tag="540" ind1=" " ind2=" ">
    <subfield code="a">License Not Specified</subfield>
  </datafield>
  <datafield tag="650" ind1="1" ind2="7">
    <subfield code="a">cc-by</subfield>
    <subfield code="2">opendefinition.org</subfield>
  </datafield>
  <datafield tag="520" ind1=" " ind2=" ">
    <subfield code="a">&lt;p&gt;Selecting reviewers for code changes is a critical step for an efficient code review process. Recent studies propose automated reviewer recommendation algorithms to support developers in this task. However, the evaluation of recommendation algorithms, when done apart from their target systems and users (i.e., code review tools and change authors), leaves out important aspects: perception of recommendations, influence of recommendations on human choices, and their effect on user experience.&lt;/p&gt;

&lt;p&gt;This study is the first to evaluate a reviewer recommender in vivo. We compare historical reviewers and recommendations for over 21,000 code reviews performed with a deployed recommender in a company environment and set out to measure the influence of recommendations on users&amp;#39; choices, along with other performance metrics.&lt;/p&gt;

&lt;p&gt;Having found no evidence of influence, we turn to the users of the recommender. Through interviews and a survey we find that, though perceived as relevant, reviewer recommendations rarely provide additional value for the respondents. We confirm this finding with a larger study at another company. The confirmation of this finding brings up a case for more user-centric approaches to designing and evaluating the recommenders.&lt;/p&gt;

&lt;p&gt;Finally, we investigate information needs of developers during reviewer selection and discuss promising directions for the next generation of reviewer recommendation tools.&lt;/p&gt;

&lt;p&gt;Preprint: https://doi.org/10.5281/zenodo.1404814&lt;/p&gt;</subfield>
  </datafield>
  <datafield tag="773" ind1=" " ind2=" ">
    <subfield code="n">doi</subfield>
    <subfield code="i">isSupplementedBy</subfield>
    <subfield code="a">10.5281/zenodo.1404755</subfield>
  </datafield>
  <datafield tag="773" ind1=" " ind2=" ">
    <subfield code="n">doi</subfield>
    <subfield code="i">isVersionOf</subfield>
    <subfield code="a">10.5281/zenodo.1404813</subfield>
  </datafield>
  <datafield tag="024" ind1=" " ind2=" ">
    <subfield code="a">10.5281/zenodo.1404814</subfield>
    <subfield code="2">doi</subfield>
  </datafield>
  <datafield tag="980" ind1=" " ind2=" ">
    <subfield code="a">publication</subfield>
    <subfield code="b">article</subfield>
  </datafield>
</record>
639
407
views
downloads
All versions This version
Views 639639
Downloads 407407
Data volume 2.4 GB2.4 GB
Unique views 575575
Unique downloads 368368

Share

Cite as