3701455
doi
10.5281/zenodo.3701455
oai:zenodo.org:3701455
user-wahr
Baker, Matthew C.
Columbia University
Thurman, Alex
Columbia University
Burke Library New York City Religions web archive collection derivatives
Ruest, Nick
York University
info:eu-repo/semantics/openAccess
Creative Commons Attribution 4.0 International
https://creativecommons.org/licenses/by/4.0/legalcode
web archives
parquet
dataframes
Arts & Humanities
Society & Culture
Universities & Libraries
Religion
New York City religions
New York (N.Y.)
Theology
<p>Web archive derivatives of the <a href="https://archive-it.org/collections/1945">Burke Library New York City Religions</a> collection from <a href="https://archive-it.org/home/Columbia">Columbia University Libraries</a>. The derivatives were created with the <a href="https://github.com/archivesunleashed/aut/">Archives Unleashed Toolkit</a> and <a href="https://cloud.archivesunleashed.org/">Archives Unleashed Cloud</a>.</p>
<p>The <strong>cul-1945-parquet.tar.gz</strong> derivatives are in the <a href="https://parquet.apache.org/">Apache Parquet format</a>, which is a <a href="http://en.wikipedia.org/wiki/Column-oriented_DBMS">columnar storage</a> format. These derivatives are generally small enough to work with on your local machine, and can be easily converted to Pandas DataFrames. See <a href="https://github.com/archivesunleashed/notebooks/blob/master/datathon-nyc/parquet_pandas_stonewall.ipynb">this</a> notebook for examples.</p>
<p><strong>Domains</strong></p>
<pre><code class="language-java">.webpages().groupBy(ExtractDomainDF($"url").alias("url")).count().sort($"count".desc)</code></pre>
<p>Produces a DataFrame with the following columns:</p>
<ul>
<li>domain</li>
<li>count</li>
</ul>
<p><strong>Web Pages</strong></p>
<pre><code class="language-java">.webpages().select($"crawl_date", $"url", $"mime_type_web_server", $"mime_type_tika", RemoveHTMLDF(RemoveHTTPHeaderDF(($"content"))).alias("content"))</code></pre>
<p>Produces a DataFrame with the following columns:</p>
<ul>
<li>crawl_date</li>
<li>url</li>
<li>mime_type_web_server</li>
<li>mime_type_tika</li>
<li>content</li>
</ul>
<p><strong>Web Graph</strong></p>
<pre><code class="language-java">.webgraph()</code></pre>
<p>Produces a DataFrame with the following columns:</p>
<ul>
<li>crawl_date</li>
<li>src</li>
<li>dest</li>
<li>anchor</li>
</ul>
<p><strong>Image Links</strong></p>
<pre><code class="language-java">.imageLinks()</code></pre>
<p>Produces a DataFrame with the following columns:</p>
<ul>
<li>src</li>
<li>image_url</li>
</ul>
<p><a href="https://github.com/archivesunleashed/aut-docs/blob/master/current/binary-analysis.md#binary-analysis"><strong>Binary Analysis</strong></a></p>
<ul>
<li>Images</li>
<li>PDFs</li>
<li>Presentation program files</li>
<li>Spreadsheets</li>
<li>Text files</li>
<li>Word processor files<br>
</li>
</ul>
<p>The <strong>cul-1945-auk.tar.gz </strong>derivatives<strong> </strong>are the <a href="https://cloud.archivesunleashed.org/derivatives">standard set of web archive derivatives</a> produced by the Archives Unleashed Cloud.</p>
<ul>
<li><strong>Gephi </strong>file, which can be loaded into <a href="https://gephi.org/">Gephi</a>. It will have basic characteristics already computed and a basic layout.</li>
<li><strong>Raw Network</strong> file, which can also be loaded into <a href="https://gephi.org/">Gephi</a>. You will have to use that network program to lay it out yourself.</li>
<li><strong>Full text</strong> file. In it, each website within the web archive collection will have its full text presented on one line, along with information around when it was crawled, the name of the domain, and the full URL of the content.</li>
<li><strong>Domains count</strong> file. A text file containing the frequency count of domains captured within your web archive.</li>
</ul>
Zenodo
2020-03-09
info:eu-repo/semantics/other
3701454
user-wahr
1583781655.586368
20360988974
md5:d36351202c8fc7e32f8e6a3428b39c86
https://zenodo.org/records/3701455/files/cul-1945-parquet.tar.gz
15947225804
md5:75a4a021e2bd261a971fae1ef4bd4092
https://zenodo.org/records/3701455/files/cul-1945-auk.tar.gz
public
10.5281/zenodo.3701454
isVersionOf
doi