Using App Reviews for Competitive Analysis: Tool Support
1. Does the paper propose a new opinion mining approach?
Yes
2. Which opinion mining techniques are used (list all of them, clearly stating their name/reference)?
REVSUM - Review Classification and Feature Extraction: Stanford CoreNLP, NLTK, Faiz A. Shah, Kairit Sirts, and Dietmar Pfahl. 2018. Simple App Review Classification with Only Lexical Features. In Proceedings of the 13th Int’l Conference on Software Technologies. SciTePress, 112–119. - sentiment analysis: Standford CoreNLP (used the review ratings to adjust the predicted sentiment score when the score predicted by the CoreNLP fully contradicts with the review rating score)
3. Which opinion mining approaches in the paper are publicly available? Write down their name and links. If no approach is publicly available, leave it blank or None.
Standford CoreNLP, NLTK Not the new technique they proposed
4. What is the main goal of the whole study?
Automatic extraction and summarization of important information such as feature evaluation, bug report and feature request from app reviews.
5. What the researchers want to achieve by applying the technique(s) (e.g., calculate the sentiment polarity of app reviews)?
REVSUM: automatically identifies developer-relevant information from reviews, such as reported bugs or requested features
6. Which dataset(s) the technique is applied on?
most recent 400 reviews for selected apps
7. Is/Are the dataset(s) publicly available online? If yes, please indicate their name and links.
No
8. Is the application context (dataset or application domain) different from that for which the technique was originally designed?
No, but verified in previous studies and some adjustment
9. Is the performance (precision, recall, run-time, etc.) of the technique verified? If yes, how did they verify it and what are the results?
Partially yes, verified in this study for sentiment analysis and in other studies for other components: - Faiz A. Shah, Kairit Sirts, and Dietmar Pfahl. 2018. Simple App Review Classification with Only Lexical Features. In Proceedings of the 13th Int’l Conference on Software Technologies. SciTePress, 112–119. - Faiz A. Shah, Kairit Sirts, and Dietmar Pfahl. 2019. Is the SAFE Approach Too Simple for App Feature Extraction? A Replication Study. In Int’l Working Conference on Requirements Engineering: Foundation for Software Quality. Springer, 21–36. https://doi.org/10.1007/978-3-030-15538-4 - Sentiment analysis: 100 randomly selected review sentences, using human labels as ground truth, the estimated accuracy of the tool was 71%. most of the 19% of the wrongly predicted cases were because of the tool’s confusion between the neutral and positive classes.
10. Does the paper replicate the results of previous work? If yes, leave a summary of the findings (confirm/partially confirms/contradicts).
No
11. What success metrics are used?
accuracy of the tool (percentage)
12. Write down any other comments/notes here.
-