There is a newer version of this record available.

Software Open Access

The TSL machine: parser, lemma analysis, sentiment analysis and autocoding for Telegram chats

Giovanni Spitale; Federico Germani; Nikola Biller - Andorno


Dublin Core Export

<?xml version='1.0' encoding='utf-8'?>
<oai_dc:dc xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:oai_dc="http://www.openarchives.org/OAI/2.0/oai_dc/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.openarchives.org/OAI/2.0/oai_dc/ http://www.openarchives.org/OAI/2.0/oai_dc.xsd">
  <dc:creator>Giovanni Spitale</dc:creator>
  <dc:creator>Federico Germani</dc:creator>
  <dc:creator>Nikola Biller - Andorno</dc:creator>
  <dc:date>2021-09-28</dc:date>
  <dc:description>The purpose of this tool is performing NLP analysis on Telegram chats. Telegram chats can be exported as .json files from the official client, Telegram Desktop (v. 2.9.2.0). 

The files are parsed, the content is used to populate a message dataframe, which is then anonymized. 

The software calculates and displays the following information:


	user count (n of users, new users per day, removed users per day);
	message count (n and relative frequency of messages, messages per day);
	autocoded messages (anonymized message dataframe with code weights assigned to each message based on a customizable set of regex rules);
	prevalence of codes (n and relative frequency);
	prevalence of lemmas (n and relative frequency);
	prevalence of lemmas segmented by autocode (n and relative frequency);
	mean sentiment per day;
	mean sentiment segmented by autocode.


The software outputs:


	messages_df_anon.csv - an anonymized file containing the progressive id of the message, the date, the univocal pseudonym of the sender, and the text;
	usercount_df.csv - user count dataframe;
	user_activity_df.csv - user activity dataframe;
	messagecount_df.csv - message count dataframe;
	messages_df_anon_coded.csv - an anonymized file containing the progressive id of the message, the date, the univocal pseudonym of the sender, the text, the codes, and the sentiment;
	autocode_freq_df.csv - general prevalence of codes;
	lemma_df.csv - lemma frequency;
	autocode_freq_df_[rule_name].csv - lemma frequency in coded messages, one file per rule;
	daily_sentiment_df.csv - daily sentiment;
	sentiment_by_code_df.csv - sentiment segmented by code;
	messages_anon.txt - anonymized text file generated from the message data frame, for easy import in other software for text mining or qualitative analysis;
	messages_anon_MaxQDA.txt - anonymized text file generated from the message data frame, formatted specifically for MaxQDA (to track speakers and codes).


Dependencies:


	pandas (1.2.1)
	json
	random
	os
	re
	tqdm (4.62.2)
	datetime (4.3)
	matplotlib (3.4.3)
	Spacy (3.1.2) + it_core_news_md
	wordcloud (1.8.1)
	Counter
	feel_it (1.0.3)
	torch (1.9.0)
	numpy (1.21.1)
	transformers (4.3.3)


This code is optimized for Italian. 

Lemma analysis is based on spaCy, which provides several other models for other languages ( https://spacy.io/models ) so it can easily be adapted.

Sentiment analysis is performed using FEEL-IT: Emotion and Sentiment Classification for the Italian Language (Kudos to Federico Bianchi &lt;f.bianchi@unibocconi.it&gt;; Debora Nozza &lt;debora.nozza@unibocconi.it&gt;; and Dirk Hovy &lt;dirk.hovy@unibocconi.it&gt;). Their work is specific for Italian. To perform sentiment analysis in other languages one could consider nltk.sentiment

The code is structured in a Jupyter-lab notebook, heavily commented for future reference.</dc:description>
  <dc:identifier>https://zenodo.org/record/5533907</dc:identifier>
  <dc:identifier>10.5281/zenodo.5533907</dc:identifier>
  <dc:identifier>oai:zenodo.org:5533907</dc:identifier>
  <dc:language>eng</dc:language>
  <dc:relation>doi:10.5281/zenodo.5533906</dc:relation>
  <dc:rights>info:eu-repo/semantics/openAccess</dc:rights>
  <dc:rights>https://creativecommons.org/licenses/by/4.0/legalcode</dc:rights>
  <dc:subject>natural language processing</dc:subject>
  <dc:subject>NLP</dc:subject>
  <dc:subject>telegram</dc:subject>
  <dc:subject>covid-19</dc:subject>
  <dc:subject>social listening</dc:subject>
  <dc:subject>green pass</dc:subject>
  <dc:subject>vaccine</dc:subject>
  <dc:subject>freedom</dc:subject>
  <dc:subject>ethics</dc:subject>
  <dc:title>The TSL machine: parser, lemma analysis, sentiment analysis and autocoding for Telegram chats</dc:title>
  <dc:type>info:eu-repo/semantics/other</dc:type>
  <dc:type>software</dc:type>
</oai_dc:dc>
290
17
views
downloads
All versions This version
Views 29015
Downloads 172
Data volume 63.4 MB11.8 MB
Unique views 26515
Unique downloads 152

Share

Cite as