{ "nbformat": 4, "nbformat_minor": 0, "metadata": { "colab": { "name": "DisboardScraper-AnalysisTools.ipynb", "provenance": [], "collapsed_sections": [ "tJttIlnLN61B", "4H3AFJH4bhDg", "iUqp7H9rFwDZ", "OkKUT5XrF969", "VICB829m17sE", "Tvy7CuAtj6LE", "GieAwY_fEaXl" ] }, "kernelspec": { "name": "python3", "display_name": "Python 3" }, "language_info": { "name": "python" } }, "cells": [ { "cell_type": "markdown", "metadata": { "id": "ncnChyhfwxA4" }, "source": [ "\n", "\n", "\n", "# Disboard Scraper and Analysis Notebook\n", "#### By PS Berge\n", "🐦 [@theiceberge](https://twitter.com/theiceberge)      ✉ [hello@psberge.com](mailto:hello@psberge.com)       [IceBerge#0666](https://darcmode.org/invite)\n", "\n", "> The Disboard Scraper and Analysis Notebook is a research toolkit for Internet scholars interested in examining networks of Discord servers. It includes tools for collecting and analyzing data from [Disboard.org](https://disboard.org/). This is a Google Colab version of the [Modified Disboard Scraper](https://https://github.com/IceBerge421/Modified-Disboard-Scraper), a fork of [DisboardScraper](https://github.com/DiscordFederation/DisboardScraper) by DiscordFederation.\n", "\n", "\n" ] }, { "cell_type": "markdown", "source": [ "# Terms of Use\n", "\n", "These are the terms of use, license, and acknowledgement information for the notebook. By using or remixing the notebook, you're agreeing to these!" ], "metadata": { "id": "tJttIlnLN61B" } }, { "cell_type": "markdown", "source": [ "## License and Conditions\n", "\n", "This code contains a public, updated release of many of the tools used in [\"Mapping Discord's Darkside: Distributed Hate Networks on Disboard\"](https://doi.org/10.1177/14614448211062548) (Heslep & Berge, 2021). The code is licensed under the [MIT License](https://github.com/IceBerge421/Modified-Disboard-Scraper/blob/master/LICENSE) as written by the original authors, meaning you are welcome to use and adapt it for your own research purposes. If you use this notebook to develop your own work, all I ask is that:\n", "\n", "1. Include attribution to \"PS Berge\" by citing this Colab (doi: 10.5281/zenodo.7305670) and/or [my study with Daniel Heslep](https://doi.org/10.1177/14614448211062548) in your work! Likewise, if you do use these tools, please let me know: I would love to hear about it! ✨ \n", "\n", "2. If you are using these tools, I expect that your work will not reinforce hegemonic systems of oppression but will rather expose those systems and help hold platforms accountable. I suggest you familiarize yourself with digital ethics (the [AoIR ethics guidelines](https://aoir.org/ethics/) are a good place to start) as well as the important work of cyberfeminist and digital humanist scholars. Additional resources can be found in the resources section of [the D/ARC community server](https://darcmode.org/invite). " ], "metadata": { "id": "I0TRO5w9SdQa" } }, { "cell_type": "markdown", "source": [ "## Acknowledgements\n", "\n", "This Colab also contains a small suite of analysis tools for quickly getting familiar with your Disboard data as well as some visualziation tools. Many of the scripts in notebook are adapted from:\n", "\n", "* Computational text analysis tools wonderfully tutorialized by [The Data Sitter's Club](https://datasittersclub.github.io/site/index.html). They are rockstars, and you should definitely check out their work if you do digital humanist work!\n", "* Notebooks built in Dr. [Anastasia Salter's Design and Development course](https://hcommons.org/members/anasalter/). I am so grateful to them for their expertise and generosity-- they taught me everything I know about Python.\n", "* [*Humanities Data Analysis: Case Studies With Python*](https://www.humanitiesdataanalysis.org/) by Folgert Karsdorp, Mike Kestemont, and Allen Riddell which **has just been updated and released open source** (2022)!" ], "metadata": { "id": "t2ojoU19SiTV" } }, { "cell_type": "markdown", "source": [ "## Join the Discord Academic Research Community\n", "\n", "These tools have been adapted to Disboard research by me, so feel free to [DM me if you encounter any problems](https://twitter.com/theiceberge). That said, if you are an academic studying Discord, **you should consider [joining D/ARC](https://darcmode.org/invite)**, the Discord Academic Research Community. In the D/ARC server, you can find more information about scraping Disboard as well as all kinds of resources (a curated Zotero library, tools, links, and more)! I've also written **a whole [post on working with Disboard data for the D/ARC blog](https://darcmode.org/scraping-disboard/)**!\n", "\n", "[](https://darcmode.org/invite)" ], "metadata": { "id": "jsGewtc3Slnx" } }, { "cell_type": "markdown", "metadata": { "id": "4H3AFJH4bhDg" }, "source": [ "# Getting Started!\n", "\n", "## Not sure where to begin? Unhide the cells below to get the gist!" ] }, { "cell_type": "markdown", "source": [ "### Get Familiar With Disboard\n", "\n", "This scraper pulls and analyzes data from https://disboard.org/, so if you're not familiar with Disboard, that's a good place to start. Disboard uses tag-based searches to network Discord communities. It, itself, is not affiliated with Discord-- but is integrated through a third-party bot. That said, Disboard is one of the largest server bulletins available (with over 1.2 million networked servers) and so it provides an incredible window into Discord communities and discourse. \n", "\n", "> Note: This toolkit includes instructions and modules for collecting Disboard data. If you want to learn more about using Disboard to perform Discord research, [check out my paper with Daniel Heslep in *New Media and Society*](https://doi.org/10.1177/14614448211062548) and my [post on the D/ARC Blog](https://darcmode.org/scraping-disboard/)." ], "metadata": { "id": "AlIRAoGER5Ss" } }, { "cell_type": "markdown", "source": [ "### Find the Tags You Want to Search\n", "\n", "Using Disboard's tag-based search, you can quickly identify the terms you might want to search for. Remember, you will often find more relevant tags to your study by scraping some preliminary data, so don't hesitate to embrace the recursive process of scraping. " ], "metadata": { "id": "WXHI1dAqSGsu" } }, { "cell_type": "markdown", "source": [ "### Note to First-Time Users\n", "\n", "> *REMEMBER: FOR MOST OF THE ANALYSIS TOOLS IN THIS NOTEBOOK, YOU DO NOT NEED TO ALTER THE DEFAULT VALUES-- THE CELLS CAN BE RUN AS THEY ARE! IF YOU ARE NEW TO DATA SCRAPING OR DIGITAL HUMANIST ANALYSIS, I RECOMMEND STARTING WITH THE DEFAULTS DURING YOUR FIRST RUN!*\n", "\n", "💖 *That's it! Good luck!* 💖" ], "metadata": { "id": "3p58uaZqSDRe" } }, { "cell_type": "markdown", "metadata": { "id": "iUqp7H9rFwDZ" }, "source": [ "# 1 Set Up The Scraper\n", "\n", "**Step 1:** Begin by **copying this Colab.** Click the `Copy to Drive` button above or go to `File` > `Save a Copy` in Drive. Note that you can run all the cells in this Gist without copying the Colab, but your changes will not be saved without first copying the Colab.\n", "\n", "**Step 2: Run the cells below to set up the scraper.** Note that it will create a new folder on your Google Drive for the cloned repository and output CSV files. \n", "\n", "> **Returning User?** *If you have used the scraper before, note that you only need to run cell 1b!*\n", "\n", "### FAQs:\n", "\n", "
\n", " Google Colab is asking for access to my Google Drive?\n", "Let it! This is only to store the CSV files you generate to a DisboardScraper folder. The code will keep all the files you generate in a single folder in your Google Drive for easy access, even if the notebook is closed. :)\n", "
\n", "\n", "
\n", " Where can I find the install directory for the scraper?\n", "Once you have set up the scraper, you can find all files used by this Colab in your `MyDrive` > `DisboardScraper` > `Modified-Disboard-Scraper` folder on Google Drive! These files will persist even once you close this notebook.\n", "
\n", "
" ] }, { "cell_type": "code", "metadata": { "id": "DwxT2gxa4hUz", "cellView": "form" }, "source": [ "#@title 1a One-Time Setup\n", "\n", "#@markdown > ***Run this cell if it is your first time using the notebook on this Google account!***\n", "\n", "#@markdown Execute this cell by pressing the _Play_ button \n", "#@markdown on the top left. You should only need to run this once.\n", "#@markdown This will clone the repository and set up the folder structure in your Google Drive. \n", "\n", "from google.colab import drive\n", "drive.mount('/content/drive')\n", "\n", "%cd /content/\n", "\n", "import os\n", "\n", "path = \"drive/MyDrive/DisboardScraper\"\n", "os.mkdir(path)\n", "\n", "% cd \"drive/MyDrive/DisboardScraper\"\n", "\n", "!git clone https://github.com/IceBerge421/Modified-Disboard-Scraper.git\n" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "cellView": "form", "id": "jDgqJNjCCpI_" }, "source": [ "#@title 1b Install Modules\n", "\n", "#@markdown > ***Run this every time you start a new session with the notebook. If you ever get an error that says `Module Not Found` you probably need to run this cell again!***\n", "\n", "#@markdown This will install the modules that Colab needs to run the scraper. If you get an error that a module is already installed, don't sweat it!\n", "\n", "from google.colab import drive\n", "import os\n", "drive.mount('/content/drive')\n", "%cd /content/\n", "\n", "!pip install cloudscraper\n", "!pip install pandas\n", "!pip install discord.py\n", "!pip install toml\n", "!pip install beautifulsoup4\n", "!pip install schema\n", "!pip install tqdm\n", "!pip install lxml\n", "!pip install nltk" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "OkKUT5XrF969" }, "source": [ "# 2 Scrape Your Data\n", "\n", "Run these cells to configure your query and collect your data!\n", "\n", "The subsequent cells will create, store, and let you download a newly-made CSV file. (All CSV files will be available in the DisboardScraper folder on Google Drive even after this notebook is closed).\n", "\n", "> **Important Update on Page Counts:** As of September, 2022, Disboard will only display the first 30 pages of results for any given tag. Unfortunately, there is not currently a workaround for this. As such **`page_count` must not exceed 30 pages or the scraper will crash.** If you want to collect larger samples, I reccomend snowballing big tags to identify smaller tag clusters, and then merging your files down the line.\n", "\n", "> **Note on Repeated Scrapes:** *If you scrape the same tag multiple times, the script **will overwrite your previous CSV file**. To preserve your data, I recommend downloading and/or renaming your CSV files often.*" ] }, { "cell_type": "code", "metadata": { "id": "aUbnSCoL6o5G", "cellView": "form" }, "source": [ "#@title 2a Run the Scraper\n", "\n", "#@markdown ### Run this cell to collect data from Disboard!\n", "\n", "#@markdown To run the scraper, configure the following values and press *Play*. The scraper will update its progress in the output field. Note that the scraper takes ~5 seconds to scrape each page, so high volume scrapes will take some time. \n", "\n", "#@markdown > Note that `page_count` cannot exceed 30 pages as of September, 2022!\n", "\n", "#@markdown ---\n", "\n", "#@markdown
Parameter Details\n", "#@markdown
tag: The term you want to search and scrape Disboard for. Note that the tag must be written as it appears on Disboard. Remember to use dashes, not spaces! (i.e. \"among-us\" not \"among us\"). Tags are not case sensititve.
\n", "#@markdown
page_count: The number of pages you want to scrape. \n", "#@markdown If you want to scrape all pages, you must look at \n", "#@markdown Disboard.org and determine the total number of \n", "#@markdown pages available manually. To quickly count the number of pages, divide \n", "#@markdown the total number of servers by 24 (the number of servers per page). \n", "#@markdown You can then check that page number by navigating to the url for the \n", "#@markdown tag and adding the page number to the end (i.e. *disboard.org/servers/tag/gaming/[page#]* ). \n", "#@markdown
\n", "\n", "%cd /content/drive/MyDrive/DisboardScraper/Modified-Disboard-Scraper\n", "\n", "tag = 'gaming' #@param {type:\"string\"}\n", "page_count = 10 #@param {type:\"integer\"}\n", "\n", "\n", "output = \"[bot]\\n\" + \"tag = \" + \"\\\"\" + tag + \"\\\"\" + \"\\n\" + \"pages = \" + str(page_count) + '''\n", "top_positional_tags = true\n", "tpt_limit = 5\n", "csv = true\n", "debug = false\n", "'''\n", "\n", "with open('/content/drive/MyDrive/DisboardScraper/Modified-Disboard-Scraper/config.toml', 'w') as writefile:\n", " writefile.write(output)\n", "\n", "!python3 app.py\n", "\n" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "r4FZd1KgDbHA", "cellView": "form" }, "source": [ "#@title 2b Download the CSV File\n", "#@markdown **Run this to download a CSV file of the last scrape you ran.**\n", "\n", "#@markdown > Note: The file will already be available in your `DisboardScraper` folder\n", "#@markdown > under the filename `[tag-word]_servers.csv`\n", "\n", "from google.colab import files\n", "\n", "csv_path = \"/content/drive/MyDrive/DisboardScraper/Modified-Disboard-Scraper/\" + tag + \"_servers.csv\"\n", "\n", "files.download(csv_path)" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "VICB829m17sE" }, "source": [ "# 3 Load Data for Analysis\n", "\n", "**Both cells must be run before proceeding to Section 4!**\n", "\n", "This module will prep your data for all of the analysis tools. You must upload the CSV file you want to analyze to the Colab or choose one from your Drive folder to analyze and import the required libraries.\n", "\n", "> **Combining Multiple Files:** If you want to examine data from multiple CSV files, you must combine them. You can quickly merge CSV files through [online mergers](http://merge-csv.com/) or in a program like Excel. Be sure to check 'keep header only in first file' when merging CSV files!" ] }, { "cell_type": "code", "metadata": { "cellView": "form", "id": "j-7DSTyF3yYb" }, "source": [ "import pandas as pd\n", "\n", "from google.colab import files\n", "import io\n", "\n", "from google.colab import data_table\n", "data_table.enable_dataframe_formatter()\n", "\n", "#@title 3a Load CSV Data (Required!)\n", "\n", "#@markdown Use this cell to load in your CSV file. Whatever file you have loaded here will be the one used for all subsequent analysis modules. You can always switch which CSV file you're analyzing by running this cell.\n", "\n", "#@markdown > **From File:** Select `Uploaded File` and run the cell. Then select the .CSV file from your desktop.\n", "\n", "#@markdown > **From Google Drive:** Set the name of the file in the project folder that you wish to load as `CSV_filename`.\n", "\n", "#@markdown
\n", "#@markdown
Parameter Details\n", "#@markdown
retreive_by: You will need to choose how you want to retrive your file. There are two methods: if you have downloaded a CSV file that you want to analyze, you can upload it by selecting \"Uploaded File\" and running the cell. This is helpful if you have collected your data elsewhere, or have downloaded the CSV files to merge them.

Otherwise, if you have a file in Google Drive, ensure that it is in the following folder: \"DisboardScraper/Modified-Disboard-Scraper/\". From there, select \"From Drive Folder\" and then enter the name of the file in CSV_filename.
\n", "#@markdown
CSV_filename (Google Drive File Only): This is only relevant if you have selected \"From Drive Folder\". Check the name of your file as it appears in MyDrive > DisboardScraper > Modified-Disboard-Scraper and enter the filename below. The notebook will then load this file in directly from your Drive folder. Filenames are generally stored as '[tag]_servers.csv'.
\n", "#@markdown
\n", "\n", "#@markdown #### How will your data be loaded?\n", "\n", "retrieve_by = \"Uploaded File\" #@param [\"Uploaded File\", \"From Drive Folder\"] {type:\"string\"}\n", "\n", "#@markdown ---\n", "\n", "#@markdown #### Google Drive files only:\n", "\n", "CSV_filename = \"[tag]_servers.csv\" #@param {type:\"string\"}\n", "\n", "filename = CSV_filename[:len(CSV_filename)-4]\n", "print(filename)\n", "\n", "if retrieve_by == \"Uploaded File\":\n", " uploaded = files.upload()\n", " filename = next(iter(uploaded))\n", " df = pd.read_csv(io.BytesIO(uploaded[filename]))\n", "else:\n", " csv_path = \"/content/drive/MyDrive/DisboardScraper/Modified-Disboard-Scraper/\" + CSV_filename\n", " df = pd.read_csv(csv_path)\n", "\n", "print(str(len(df)) + \" servers loaded. Data preview: \")\n", "\n", "df.head(len(df))\n", "\n" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "source": [ "#@title 3b Import Dependencies (Required!)\n", "\n", "#@markdown Run this cell to import all the required libraries for the following \n", "#@markdown analysis tools. Once this has been run, you'll be good to dive into analysis!\n", "\n", "#@markdown > Note: You may get a warning after running this cell that says \"UserWarning: The twython library has not been installed.\" You can safely disregard this error, as we are not using any Twitter tools here. :)\n", "\n", "import collections\n", "import datetime as dt\n", "from datetime import datetime\n", "from google.colab import files\n", "\n", "import matplotlib.pyplot as plt\n", "import networkx as nx\n", "import numpy as np\n", "\n", "import bokeh\n", "from bokeh.models import ColumnDataSource\n", "from bokeh.plotting import figure, show, output_file, save\n", "from bokeh.io import output_notebook\n", "from bokeh.palettes import Viridis256\n", "from bokeh.models.tools import HoverTool\n", "from bokeh.models.formatters import DatetimeTickFormatter\n", "from bokeh.models import ColorBar\n", "from bokeh.transform import linear_cmap\n", "from bokeh.models.tools import WheelZoomTool\n", "from bokeh.transform import jitter\n", "\n", "import nltk.data\n", "from nltk.sentiment.vader import SentimentIntensityAnalyzer\n", "from nltk import sentiment\n", "from nltk import word_tokenize\n", "\n", "nltk.downloader.download('vader_lexicon')\n", "\n", "from wordcloud import WordCloud, STOPWORDS\n", "import random\n", "\n", "import matplotlib.pyplot as plt\n", "from matplotlib.pyplot import figure\n", "\n", "from IPython.display import display" ], "metadata": { "cellView": "form", "id": "omtg6grDPhZR" }, "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "GjXGQVBLn10c" }, "source": [ "\n", "# 4 Analysis Tools\n", "\n", "\n", "\n", "Section 4 contains several tools for quickly analyzing your Disboard data.\n", "\n", "Beyond this point, order is not important: *feel free to jump around between the different tools here.* The only required step is that you *must run both cells in Section 3 before running anything in Analysis Tools*. \n", "\n", "\n" ] }, { "cell_type": "markdown", "source": [ "## Module Overviews\n", "\n", "There are eight different analysis and visualization tools for you to work with:\n", "\n", "* **4a View Most/Least Popular Servers:\\** Sort your server data by Members Online. Useful for finding popular (and unpopular) communities quickly. Also calculates the average number of online members across the sample. \n", "* **4b View Newest/Oldest Servers:** Sort your server data by Creation Date. Useful for quickly finding the oldest and newest communities in your dataset. \n", "* **4c View Most/Least Popular Tags:** Find the most and least prominent tags in your dataset. Useful for finding co-occurring and promintent tags to expand your sample with. This is especially helpful for building a snowball sample.\n", "* **4d View Server Activity Graph:** Show an interactive graph of Members Online and Date Created to get a sense of where communities fall in popularity and duration.\n", "* **4e Description Word Cloud:** Generate a colorful word cloud of your description data. Useful for quick surface glances and creating visualizations for presentations / papers.\n", "* **4f Examine Tag by Position:** View where in Disboard's five tag slots certain tags tend to appear. Useful for seeing how they are being prioritized by listers.\n", "* **4g Calculate Sentiment:** \n", " * *View Sentiment Scores:* Use natural language processing to get the sentiment score of each server description in the dataset. Useful for quickly identifying outliers and strongly-worded descriptions.\n", " * *View Interactive Sentiment Chart:* Visualize your larger dataset by sentiment in an interactive HTML file. Useful for diving into the data with attention to sentiment.\n", "* **4h Tag Network Graph:** The most complex (and, hopefully, helpful) visualization tool. This will create a relational network visualization of all the tags used in your dataset. Tags that commonly co-occur will be linked and more popular tags are shown as larger nodes. Two visualization options: Spring and Shell. " ], "metadata": { "id": "Idu6qDJSBts2" } }, { "cell_type": "code", "metadata": { "cellView": "form", "id": "jLFf5jlAo1x0" }, "source": [ "#@title 4a View Most/Least Popular Servers\n", "#@markdown Run this cell to view a list of the most recent or oldest servers in \n", "#@markdown your data set. Note that the datetime format for Disboard is YYYY:MM:DD.\n", "#@markdown Configure with the follwing values:\n", "\n", "#@markdown
Parameter Details\n", "#@markdown
sort_by: Select 'Most Popular' or 'Least Popular' servers.
\n", "\n", "df2 = df\n", "\n", "sort_by = 'Most Popular' #@param [\"Most Popular\", \"Least Popular\"]\n", "\n", "if sort_by == \"Least Popular\":\n", " print(\"Least Popular Servers:\")\n", " display(df2.sort_values('Members Online', ascending=True).tail(len(df2)))\n", "else:\n", " print(\"Most Popular Servers:\")\n", " display(df2.sort_values('Members Online', ascending=False).head(len(df2)))\n", "\n", " print(\"***************\\nAverage Member Count: \" + str(df2['Members Online'].mean()))\n", "print(\"Smallest: \" + str(df2['Members Online'].min()) + \" members. Largest: \" + str(df2['Members Online'].max()) + \" members.\")" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "cellView": "form", "id": "EelvMPS5FGd5" }, "source": [ "#@title 4b View Newest/Oldest Servers\n", "#@markdown Run this cell to view a list of the most recent or oldest servers in \n", "#@markdown your data set. Note that the datetime format for Disboard is YYYY:MM:DD.\n", "#@markdown Configure with the follwing values:\n", "\n", "#@markdown
Parameter Details\n", "#@markdown
sort_by: Select 'Newest' or 'Oldest' servers.
\n", "\n", "df2 = df\n", "\n", "sort_by = 'Newest' #@param [\"Oldest\", \"Newest\"]\n", "\n", "if sort_by == \"Oldest\":\n", " print(\"Oldest Servers:\")\n", " display(df2.sort_values('Creation Date', ascending=True).tail(len(df2)))\n", "else:\n", " print(\"Newest Servers:\")\n", " display(df2.sort_values('Creation Date', ascending=False).head(len(df2)))\n", "\n", "print(\"***************\\n\")\n", "print(\"Oldest server: \" + str(df2['Creation Date'].min()) + \". Newest server: \" + str(df2['Creation Date'].max()) + \".\")\n" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "cellView": "form", "id": "YjiuIZabwlzC" }, "source": [ "#@title 4c Most Popular Tags\n", "#@markdown This will let you identify the most popular co-occurring tags \n", "#@markdown within the sample. (Note that this counts across columns)!\n", "\n", "#@markdown
Parameter Details\n", "#@markdown
list_count: The number of tags to show in the list (default: 30).
\n", "#@markdown
graph_count: The number of tags to display in in the graph (default: 20).
\n", "\n", "list_count = 30 #@param {type:\"integer\"}\n", "graph_count = 20 #@param {type:\"integer\"}\n", "\n", "from matplotlib.pyplot import figure\n", "\n", "df2 = df[['Tag 1','Tag 2','Tag 3', 'Tag 4', 'Tag 5']]\n", "\n", "figure(figsize=(16, 10))\n", "\n", "df2.stack().value_counts()[0:graph_count].plot(kind=\"barh\", width = 0.6, color = \"#2b8cbe\", rot=0)\n", "\n", "display(df2.stack().value_counts().head(list_count))" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "k5QymKM1Oct5", "cellView": "form" }, "source": [ "#@title 4d View Server Activity Graph\n", "\n", "#@markdown Run this cell to view an interactive HTML graph that compares server Creation Date and Members Online. This is useful for seeing overall popularity and duration of communities in your dataset. \n", "#@markdown You can click and drag to navigate, use the WheelZoom feature to zoom in, and hovering over a point will display the server information.\n", "#@markdown You will need to set two variables:\n", "\n", "#@markdown > **sample_size:** If dealing with large data, you may want to only view a sample of servers.\n", "#@markdown You can set the sample size here. *Leave the sample size as 0 to view all servers.*\n", "\n", "#@markdown > **save_output:** If this box is ticked, the cell will save the outputted HTML file to your Disboard\n", "#@markdown > scraper folder as 'ServerActivityGraph ([filename] - [date/time run]).html'\n", "\n", "sample_size = 0 #@param {type:\"integer\"}\n", "save_output = False #@param {type:\"boolean\"}\n", "\n", "from bokeh.plotting import figure, show, output_file, save\n", "\n", "checkout_as_datetime = pd.to_datetime(df['Creation Date'], format='%Y:%m:%d')\n", "\n", "df3 = df\n", "\n", "df3['Members'] = df['Members Online']\n", "df3['Date'] = checkout_as_datetime.dt.date\n", "df3['CalDate'] = df['Creation Date']\n", "df3['Name'] = df3['Server Name']\n", "\n", "if sample_size == 0:\n", " sample = df3.sample(len(df3))\n", "else:\n", " sample = df3.sample(sample_size)\n", "\n", "source = ColumnDataSource(sample)\n", "\n", "p = figure(plot_width=1000)\n", "p.circle(x='Date', y= 'Members Online',\n", " source=source,\n", " size=10, color='green')\n", "\n", "p.title.text = 'Server Activity'\n", "p.yaxis.axis_label = 'Members Online'\n", "p.xaxis.axis_label = 'Date'\n", "\n", "p.yaxis.formatter.use_scientific = False\n", "\n", "p.xaxis.formatter=DatetimeTickFormatter(days=\"%m/%d %H:%M\",\n", "months=\"%m/%d %H:%M\",\n", "hours=\"%m/%d %H:%M\",\n", "minutes=\"%m/%d %H:%M\")\n", "\n", "hover = HoverTool()\n", "hover.tooltips=[\n", " ('Name', '@Name'),\n", " ('Date', '@CalDate'),\n", " ('Description', '@Description'),\n", " ('Members', '@Members')\n", "]\n", "\n", "p.add_tools(hover)\n", "\n", "output_notebook()\n", "\n", "show(p)\n", "\n", "if save_output:\n", " now = datetime.now()\n", " dt_string = now.strftime(\"%d-%m-%Y-%H-%M-%S\")\n", "\n", " %cd /content/drive/MyDrive/DisboardScraper/Modified-Disboard-Scraper\n", "\n", " out_str = \"ServerActivityGraph (\" + filename + \" - \" + dt_string + \").html\"\n", " disp_str = \"Server Activity\" + filename\n", "\n", " bokeh.plotting.output_file(out_str)\n", " bokeh.io.save(\n", " p, \n", " title= disp_str\n", " )" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "cellView": "form", "id": "nI0ARrTDz6Y0" }, "source": [ "#@title 4e Generate Description Word Cloud\n", "\n", "#@markdown This is a simple module that will generate colorful wordclouds from your data. Useful for creating easy graphics for a paper or presentation, or for getting a sense of some of the most frequent words in your sample.\n", "\n", "#@markdown Note that this cloud does not pull from tags, only the descriptions. \n", "#@markdown You can configure the following values:\n", "\n", "#@markdown > **additional_stopwords**: add words here (seperated by spaces) that \n", "#@markdown you want to remove from the dataset. The words are not case sensitive.\n", "\n", "#@markdown > **debug_stopwords**: You can check this to print out all stopwords\n", "#@markdown currently being applied to the wordcloud. Check this if you're having\n", "#@markdown trouble with the stopwords, or if you aren't sure what's being rewoved.\n", "\n", "#@markdown > **save_output**: If this box is ticked, the cell will save the outputted HTML file to your Disboard\n", "#@markdown scraper folder as 'WordCloud ([filename] - [date/time run]).png'. Note that the file will take a moment to appear in the folder.\n", "\n", "additional_stopwords = \"community server join discord member\" #@param {type:\"string\"}\n", "debug_stopwords = False #@param {type:\"boolean\"}\n", "save_output = False #@param {type:\"boolean\"}\n", "#@markdown ---\n", "\n", "#@markdown #### Adjust Color Values (Single Gradient Mode Only)\n", "\n", "#@markdown You can modify the coloration of the word cloud with the colors below. You can preview your color options using this tool.\n", "\n", "#@markdown > **single_gradient**: Check this box to enable single gradient mode and use the sliders below to adjust the coloration. When unchecked, you will get the default rainbow wordcloud.\n", "\n", "#@markdown > **hue**: Set the primary color for the cloud using HSL. The hue can be any number between 0 and 359 degrees. \n", "\n", "#@markdown > **max_light**: Sets the brightness limit for words (recommended: 90). *Must not be lower than min_dark!*\n", "\n", "#@markdown > **min_dark**: Set the darkest limit for words (recommended: 40). *Must not be higher than max_light!*\n", "\n", "from matplotlib.pyplot import figure\n", "\n", "single_gradient = False #@param {type:\"boolean\"}\n", "hue = 0 #@param {type:\"slider\", min:0, max:359, step:1}\n", "max_light = 90 #@param {type:\"slider\", min:0, max:100, step:1}\n", "min_dark = 40 #@param {type:\"slider\", min:0, max:100, step:1}\n", "\n", "newstops = additional_stopwords.split()\n", "stop_words = newstops + list(STOPWORDS)\n", "\n", "if debug_stopwords:\n", " print(\"Added Stopwords: \" + str(newstops))\n", " print(stop_words)\n", "\n", "if single_gradient == False:\n", " def color_func(word, font_size, position, orientation, random_state=None,**kwargs):\n", " return \"hsl(%s, 80%%, %d%%)\" % (random.randint(0,360), random.randint(60, 90))\n", "else:\n", " \n", " def color_func(word, font_size, position, orientation, random_state=None,**kwargs):\n", " return \"hsl(%s, 80%%, %d%%)\" % (hue, random.randint(min_dark, max_light))\n", "\n", "string = pd.Series(df['Description']).str.cat(sep=' ')\n", "wordcloud = WordCloud(width=1600, stopwords=stop_words,height=800,max_font_size=200,max_words=50,collocations=False, background_color='white').generate(string)\n", "wordcloud.recolor(color_func=color_func, random_state=3)\n", "plt.figure(figsize=(40,30))\n", "plt.imshow(wordcloud, interpolation=\"bilinear\")\n", "plt.axis(\"off\")\n", "plt.show()\n", "\n", "if save_output:\n", " now = datetime.now()\n", " dt_string = now.strftime(\"%d-%m-%Y-%H-%M-%S\")\n", "\n", " %cd /content/drive/MyDrive/DisboardScraper/Modified-Disboard-Scraper\n", "\n", " out_str = \"WordCloud (\" + filename + \" - \" + dt_string + \").png\"\n", " plt.savefig(out_str)\n" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "pWRVMbk4AiG2", "cellView": "form" }, "source": [ "#@title 4f Create Positional Tag Graph\n", "\n", "#@markdown This module will let you view a tag's frequency by position. Disboard, by default, uses a five-tag system as inputted by the user. This can be useful for determining how listers are prioritizing certain tags over others.\n", "#@markdown Enter a tag and run the cell to generate a graph of position frequency.\n", "\n", "#@markdown > Note: The **tag** value must be a token that appears in the dataset, and should not include spaces.\n", "\n", "pd.options.mode.chained_assignment = None # default='warn'\n", "\n", "from matplotlib.pyplot import figure\n", "\n", "tag = 'giveaway' #@param {type:\"string\"}\n", "\n", "def TagPosition(word):\n", " plotter = pd.DataFrame({'Tag Position':[\"Tag 1\", \"Tag 2\", \"Tag 3\", \"Tag 4\", \"Tag 5\"], 'Count':[0,0,0,0,0]})\n", "\n", " gc1 = df.groupby('Tag 1')['Server Name'].count()\n", " gc2 = df.groupby('Tag 2')['Server Name'].count()\n", " gc3 = df.groupby('Tag 3')['Server Name'].count()\n", " gc4 = df.groupby('Tag 4')['Server Name'].count()\n", " gc5 = df.groupby('Tag 5')['Server Name'].count()\n", "\n", " if word in gc1:\n", " plotter['Count'][0] = gc1[word]\n", " if word in gc2:\n", " plotter['Count'][1] = gc2[word]\n", " if word in gc3:\n", " plotter['Count'][2] = gc3[word]\n", " if word in gc4:\n", " plotter['Count'][3] = gc4[word]\n", " if word in gc5:\n", " plotter['Count'][4] = gc5[word]\n", "\n", " ax = plotter.plot.barh(x = 'Tag Position', y = 'Count', width = 0.6, color = \"#2b8cbe\", rot=0)\n", "\n", " for i, v in enumerate(plotter['Count']):\n", " ax.text(v + 0.5, i + .2, str(v), color = '#a6bddb', fontweight = 'bold')\n", "\n", "TagPosition(tag)" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "Tvy7CuAtj6LE" }, "source": [ "### 4g Calculate Sentiment\n", "\n", "This module will add a new column to your dataframe that calculates the sentiment of the server description using the NLTK (natural language toolkit) and Vader sentiment analyzer. Sentiment analysis is a finnicky, and often unreliable, but it can be a useful way to quickly identify outliers in your dataset. \n", "\n", "> If you want context on sentiment analysis and how it works, I reccommend reading [this issue of The Data Sitters Club](https://datasittersclub.github.io/site/dsc11.html) by Katherine Bowers and Quinn Dombrowski!" ] }, { "cell_type": "code", "metadata": { "cellView": "form", "id": "oKMOQWOckvtZ" }, "source": [ "#@title Step 1: View Sentiment Scores\n", "#@markdown Run this cell to add a sentiment column to your data and view a table\n", "#@markdown of the servers and their respective scores. You can configure two fields:\n", "#@markdown > **sort_sentiment**: Show the servers with the highest or lowest sentiment scores.\n", "\n", "sort_sentiment = 'Highest' #@param [\"Lowest\", \"Highest\"]\n", "\n", "sid = SentimentIntensityAnalyzer()\n", "\n", "def calculate_sentiment(text):\n", " # Run VADER on the text\n", " scores = sid.polarity_scores(str(text))\n", " # Extract the compound score\n", " compound_score = scores['compound']\n", " # Return compound score\n", " return compound_score\n", " \n", "df['Sentiment Score'] = df['Description'].apply(calculate_sentiment)\n", "\n", "if sort_sentiment == 'Highest':\n", " display(df.sort_values(by='Sentiment Score', ascending=False))\n", "else:\n", " display(df.sort_values(by='Sentiment Score', ascending=True))" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "cellView": "form", "id": "WDnAa-RUq0Uf" }, "source": [ "#@title Step 2: View Interactive Sentiment Chart\n", "\n", "#@markdown Run this cell to view an interactable HTML graph of the sentiment distribution of your dataset. \n", "#@markdown Note that you can use the WheelZoom feature (located to the right of the graph) to zoom in, and hovering over a point will display the server information.\n", "\n", "\n", "#@markdown You will need to set two fields:\n", "\n", "#@markdown > **xaxis**: You can sort the graph by either Creation Date or Members Online. This will let you explore how sentiment of your dataset varies across the sample in the context of either server longevity or size.\n", "\n", "#@markdown > **jitter_amount**: Jitter is used to make data visible without overlapping by moving the points tiny increments to the left or right. Control the jitter amount here, or set to 0 for no jitter.\n", "\n", "#@markdown > **save_output**: If this box is ticked, the cell will save the outputted HTML file to your Disboard\n", "#@markdown > scraper folder as 'SentimentGraph ([filename] - [date/time run]).html'. Note that the file will take a moment to appear in the folder.\n", "\n", "xaxis = 'Member_Count' #@param [\"Member_Count\", \"Date_Created\"]\n", "jitter_amount = 1 #@param {type:\"integer\"}\n", "save_output = False #@param {type:\"boolean\"}\n", "\n", "from bokeh.plotting import figure, show, output_file, save\n", "\n", "def calculate_sentiment(text):\n", " # Run VADER on the text\n", " scores = sid.polarity_scores(str(text))\n", " # Extract the compound score\n", " compound_score = scores['compound']\n", " # Return compound score\n", " return compound_score\n", " \n", "df['Sentiment Score'] = df['Description'].apply(calculate_sentiment)\n", "\n", "checkout_as_datetime = pd.to_datetime(df['Creation Date'], format='%Y:%m:%d')\n", "\n", "df4 = df\n", "\n", "df4['Members'] = df['Members Online']\n", "df4['Date'] = checkout_as_datetime.dt.date\n", "df4['CalDate'] = df['Creation Date']\n", "df4['Name'] = df['Server Name']\n", "df4['Sentiment'] = df['Sentiment Score']\n", "\n", "source = ColumnDataSource(df4)\n", "mapper = linear_cmap(field_name='Sentiment Score', palette=Viridis256 ,low=-1 ,high=1)\n", "p = figure(plot_height=1000, plot_width=1200, toolbar_location=\"right\")\n", "\n", "if xaxis == 'Member_Count':\n", " p.circle(x=jitter('Members',width=jitter_amount,range=p.x_range), y='Sentiment Score', source=source, size=5, line_color=mapper,color=mapper, fill_alpha=1)\n", " p.toolbar.active_scroll = WheelZoomTool()\n", " p.title.text = 'Server Sentiment Analysis'\n", " p.xaxis.axis_label = 'Members Online'\n", " p.yaxis.axis_label = 'Sentiment Score'\n", " p.xaxis.formatter.use_scientific = False\n", "else: \n", " p.circle(x=jitter('Date',width=jitter_amount,range=p.x_range), y='Sentiment Score', source=source, size=5, line_color=mapper,color=mapper, fill_alpha=1)\n", " p.toolbar.active_scroll = WheelZoomTool()\n", " p.title.text = 'Server Sentiment Analysis'\n", " p.xaxis.axis_label = 'Date'\n", " p.yaxis.axis_label = 'Sentiment Score'\n", " p.xaxis.formatter=DatetimeTickFormatter(days=\"%m/%d %H:%M\",\n", " months=\"%m/%d %H:%M\",\n", " hours=\"%m/%d %H:%M\",\n", " minutes=\"%m/%d %H:%M\")\n", "\n", "p.yaxis.formatter.use_scientific = False\n", "\n", "color_bar = ColorBar(color_mapper=mapper['transform'], width=8)\n", "p.background_fill_color = \"#A9B6CC\"\n", "p.add_layout(color_bar, 'right')\n", "\n", "if xaxis == 'Member_Count':\n", " hover = HoverTool()\n", " hover.tooltips= \"\"\"\n", "
@Name
\n", " Members: @Members
\n", " Description: @Description
Sentiment: @Sentiment\n", "
\n", " \"\"\"\n", "else: \n", " hover = HoverTool()\n", " hover.tooltips= \"\"\"\n", "
@Name
\n", " Date: @CalDate
\n", " Description: @Description
Sentiment: @Sentiment\n", "
\n", " \"\"\"\n", "\n", "p.add_tools(hover)\n", "\n", "output_notebook()\n", "\n", "show(p)\n", "\n", "if save_output:\n", " now = datetime.now()\n", " dt_string = now.strftime(\"%d-%m-%Y-%H-%M-%S\")\n", "\n", " %cd /content/drive/MyDrive/DisboardScraper/Modified-Disboard-Scraper\n", "\n", " out_str = \"SentimentGraph (\" + filename + \" - \" + dt_string + \").html\"\n", " disp_str = \"Sentiment Graph\" + filename\n", "\n", " bokeh.plotting.output_file(out_str)\n", " bokeh.io.save(\n", " p, \n", " title= disp_str)" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "GieAwY_fEaXl" }, "source": [ "### 4h Create Tag Network Map\n", "\n", "This module is the most complex, but perhaps the most useful in this notebook. Running this cell will generate a full network map of the tags in your sample. \n", "\n", "**Two important things:**\n", "\n", "> This visualization represents *adjacent tags*. Those with stronger connections are those are that most frequently appear together in server listings. \n", "\n", "> This will ultimately be a very large file, and may take some time to run. It's usually best if you generate the network map and then view it as a seperate file.\n", "\n", "\n", "\n" ] }, { "cell_type": "code", "metadata": { "cellView": "form", "id": "azZWwwOQErzA" }, "source": [ "#@title **Step 1:** Set Up Tag-Network Map\n", "\n", "#@markdown Run this cell to set up the network visualization. This cell itself won't \n", "#@markdown generate anything, but will tell you the number of Nodes and Edges\n", "#@markdown that the Tag-Network map will generate. Once you're done, **run the next cell.**\n", "\n", "df5 = df\n", "\n", "df5 = df5.fillna(\"Empty\")\n", "\n", "wordList = list({})\n", "\n", "for x in df5['Tag 1']:\n", " if x not in wordList and x != \"Empty\":\n", " wordList.append(x)\n", "\n", "for x in df5['Tag 2']:\n", " if x not in wordList and x != \"Empty\":\n", " wordList.append(x)\n", "\n", "for x in df5['Tag 3']:\n", " if x not in wordList and x != \"Empty\":\n", " wordList.append(x)\n", "\n", "for x in df5['Tag 4']:\n", " if x not in wordList and x != \"Empty\":\n", " wordList.append(x)\n", "\n", "for x in df5['Tag 5']:\n", " if x not in wordList and x != \"Empty\":\n", " wordList.append(x)\n", "\n", "G = nx.Graph()\n", "\n", "for ind in df5.index:\n", " tags = list({})\n", " tags.append(df5['Tag 1'][ind])\n", " tags.append(df5['Tag 2'][ind])\n", " tags.append(df5['Tag 3'][ind])\n", " tags.append(df5['Tag 4'][ind])\n", " tags.append(df5['Tag 5'][ind])\n", " for i in range(len(tags) - 1):\n", " try:\n", " tag_i = tags[i]\n", " for z in range(len(tags)-1-i):\n", " tag_j = tags[len(tags)-1-z]\n", " if G.has_edge(tag_i, tag_j):\n", " G[tag_i, tag_j]['weight'] += 1\n", " else:\n", " G.add_edge(tag_i, tag_j, weight =1)\n", " \n", " except KeyError:\n", " continue\n", "if G.has_node(\"Empty\"): \n", " G.remove_node(\"Empty\")\n", "\n", "print(f\"N nodes = {G.number_of_nodes()}, N edges = {G.number_of_edges()}\")\n" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "8zt0fxROG3L5", "cellView": "form" }, "source": [ "#@title **Step 2:** Generate Tag-Network Visualization \n", "\n", "#@markdown Use this cell to create a network graph of your dataset. Note that this can take some time and will generate a large image. \n", "\n", "#@markdown > **visualization_type:** There are two visualization types available, a **spring visualization** and a **shell visualization**.\n", "#@markdown > The spring visualization uses a traditional clustering method, while the shell visualization traces relationships across a large circle. \n", "#@markdown > If you are unsure which to use, stick with the spring visualization.\n", "\n", "#@markdown > **image_format:** Select what kind of file you want to export the visualization as. \n", "#@markdown > PNG files are more easily shared and are smaller. SVG files are 'lossless' meaning\n", "#@markdown you can zoom in as much as you want and it will still be clear (SVG is reccomended).\n", "\n", "#@markdown > **save_output:** When checked, this will save a copy of the image to your DisboardScraper folder.\n", "\n", "visualization_type = \"Spring Visualization\" #@param [\"Spring Visualization\", \"Shell Visualization\"] {type:\"string\"}\n", "\n", "image_format = \"SVG\" #@param [\"SVG\", \"PNG\"] {type:\"string\"}\n", "\n", "save_output = False #@param {type:\"boolean\"}\n", "\n", "#@markdown ---\n", "\n", "#@markdown #### General Customization Options\n", "\n", "#@markdown You can adjust the network visualization with the following variables. Note that these will not impact the values of the network, only the aesthetics of the visualization. \n", "\n", "#@markdown > **adj_figure_size:** Adjust, in *inches* the size of the created graphic. Note that creating a very large canvas will not necessarily mean a clearer graphic. Going too large may mean an unmanageably large file! (Default: 70).\n", "\n", "#@markdown > **adj_font_size:** Adjust the size of the font labels (default: 10).\n", "\n", "#@markdown > **adj_font_color:** Adjust the color of the font labels. Can be in [hexcode](https://color.adobe.com/create/color-wheel) or an established [HTML color name](https://en.wikipedia.org/wiki/Web_colors) (default: #dd1c77). \n", "\n", "#@markdown > **adj_node_alpha:** Adjust the transparency of the nodes (circles) in the visualization. The lower the value, the more transparent the nodes will be. (Default: 0.4). \n", "\n", "#@markdown > **adj_edge_alpha:** Adjust the transparency of the edges (lines) in the visualization. The lower the value, the more transparent the nodes will be. (Default: 0.2). \n", "\n", "adj_figure_size = 70 #@param {type:\"integer\"}\n", "adj_font_size = 10 #@param {type:\"integer\"}\n", "adj_font_color = \"#dd1c77\" #@param {type:\"string\"}\n", "adj_node_alpha = 0.4 #@param {type:\"slider\", min:0, max:1, step:0.01}\n", "adj_edge_alpha = 0.2 #@param {type:\"slider\", min:0, max:1, step:0.01}\n", "\n", "#@markdown ---\n", "#@markdown #### Spring Visualization Only \n", "\n", "#@markdown > **adj_springvalue:** Adjust, the 'k' or springiness value of the network. This value sets the 'optimal distance' (in inches) between nodes. Lowering the number will make the network more concentrated, while a larger number will explode it outwards. Setting the value below 0.5 may give a horizontal 'scrunching' effect. Going above 1 may make individual nodes easier to see, but relationships harder to follow (default: 0.55).\n", "\n", "adj_springvalue = 0.55 #@param {type:\"number\"}\n", "#@markdown ---\n", "\n", "from matplotlib.pyplot import figure\n", "\n", "interactions = collections.Counter()\n", "\n", "for tag_i, tag_j, data in G.edges(data=True):\n", " interaction_count = data['weight']\n", " interactions[tag_i] += interaction_count\n", " interactions[tag_j] += interaction_count\n", "\n", "nodesizes = [interactions[tag] * 4 for tag in G]\n", "\n", "pd.options.mode.chained_assignment = None \n", "fig = plt.figure(figsize=(adj_figure_size, adj_figure_size))\n", "if visualization_type == \"Spring Visualization\":\n", " pos = nx.spring_layout(G, k=adj_springvalue, iterations=200)\n", "else:\n", " pos = nx.shell_layout(G, nlist=None, rotate=None, scale=1, center=None, dim=2)\n", "nx.draw_networkx_edges(G, pos, alpha=adj_edge_alpha)\n", "nx.draw_networkx_nodes(G, pos, node_size=nodesizes, alpha=adj_node_alpha)\n", "\n", "nx.draw_networkx_labels(G, pos, font_size=adj_font_size, font_color = \"#dd1c77\")\n", "plt.axis('off')\n", "\n", "if save_output:\n", " now = datetime.now()\n", " dt_string = now.strftime(\"%d-%m-%Y-%H-%M-%S\")\n", "\n", " %cd /content/drive/MyDrive/DisboardScraper/Modified-Disboard-Scraper\n", "\n", " if image_format == \"SVG\":\n", " viz_out_str = \"NetworkMap (\" + filename + \" - \" + dt_string + \").svg\"\n", " else:\n", " viz_out_str = \"NetworkMap (\" + filename + \" - \" + dt_string + \").png\"\n", "\n", " plt.savefig(viz_out_str)\n", "\n", " #@markdown ***Click the image to zoom in! You can then pan around with the mouse wheel. For easiest viewing, download the image with the next cell!***\n", "\n" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "source": [ "#@title **Step 3:** Download Visualization \n", "\n", "#@markdown Run this cell to download your new tag network! (Only works once you've run the last two cells and chosen 'save output'!)\n", "\n", "files.download(viz_out_str)\n" ], "metadata": { "cellView": "form", "id": "Wx8qdQx6J5PY" }, "execution_count": null, "outputs": [] } ] }