Keyword frequencies in arXiv and SSRN working papers

The dataset contains the raw results of the trend analysis performed on two working paper repositories: ArXiv and SSRN.

ArXiv: the dataset consists of working papers acquired via ArXiv’s API (https://arxiv.org/help/api/index). Working papers have been collected from the Computer Science discipline (all CS categories)

SSRN (The Social Science Research Network): Working papers have been collected from two broad categories: 1. Information Systems & eBusiness, 2. Innovation. 

For each repository, there are two separate analyses: for the period 2016.01-2019.12 and for 2020.01-2020.06 (COVID-19).

Methodology:

• Frequency of appearances for all unigrams and bigrams in the texts
• Frequency: number of appearances of every term divided by the number of all terms (for every month and in case of COVID - every week due to the shorter time period) 
• Average monthly / weekly change in the analised term's frequency is calculated by OLS regressions
• The dependent variable of the estimation is the frequency index, while the number of months since the beginning of the analysed period (January 2016) is the independent variable (in the case of COVID: weeks since January 2020)
• The regression coefficient (referred to as coef) shows by how much on average the analysed expression’s frequency changed with every observed week (marginal change of the frequency), revealing which keywords had the biggest weekly growth