UPDATE: Zenodo migration postponed to Oct 13 from 06:00-08:00 UTC. Read the announcement.

Software Open Access

Word Segmentation in Sanskrit Using Energy Based Models

Amrith Krishna; Bishal Santra; Sasi Prasanth Bandaru; Gaurav Sahu; Vishnu Dutt Sharma; Pavankumar Satuluri; Pawan Goyal


DataCite XML Export

<?xml version='1.0' encoding='utf-8'?>
<resource xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://datacite.org/schema/kernel-4" xsi:schemaLocation="http://datacite.org/schema/kernel-4 http://schema.datacite.org/meta/kernel-4.1/metadata.xsd">
  <identifier identifierType="DOI">10.5281/zenodo.1035413</identifier>
  <creators>
    <creator>
      <creatorName>Amrith Krishna</creatorName>
      <affiliation>IIT Kharagpur</affiliation>
    </creator>
    <creator>
      <creatorName>Bishal Santra</creatorName>
      <affiliation>IIT Kharagpur</affiliation>
    </creator>
    <creator>
      <creatorName>Sasi Prasanth Bandaru</creatorName>
      <affiliation>IIT Kharagpur</affiliation>
    </creator>
    <creator>
      <creatorName>Gaurav Sahu</creatorName>
      <affiliation>IIT Kharagpur</affiliation>
    </creator>
    <creator>
      <creatorName>Vishnu Dutt Sharma</creatorName>
      <affiliation>American Express</affiliation>
    </creator>
    <creator>
      <creatorName>Pavankumar Satuluri</creatorName>
      <affiliation>Chinmya Visvavidyapeeth</affiliation>
    </creator>
    <creator>
      <creatorName>Pawan Goyal</creatorName>
      <affiliation>IIT Kharagpur</affiliation>
    </creator>
  </creators>
  <titles>
    <title>Word Segmentation in Sanskrit Using Energy Based Models</title>
  </titles>
  <publisher>Zenodo</publisher>
  <publicationYear>2018</publicationYear>
  <dates>
    <date dateType="Issued">2018-08-23</date>
  </dates>
  <resourceType resourceTypeGeneral="Software"/>
  <alternateIdentifiers>
    <alternateIdentifier alternateIdentifierType="url">https://zenodo.org/record/1035413</alternateIdentifier>
  </alternateIdentifiers>
  <relatedIdentifiers>
    <relatedIdentifier relatedIdentifierType="DOI" relationType="IsVersionOf">10.5281/zenodo.1035412</relatedIdentifier>
    <relatedIdentifier relatedIdentifierType="URL" relationType="IsPartOf">https://zenodo.org/communities/cnerg</relatedIdentifier>
  </relatedIdentifiers>
  <rightsList>
    <rights rightsURI="https://creativecommons.org/licenses/by/4.0/legalcode">Creative Commons Attribution 4.0 International</rights>
    <rights rightsURI="info:eu-repo/semantics/openAccess">Open Access</rights>
  </rightsList>
  <descriptions>
    <description descriptionType="Abstract">&lt;p&gt;This is the repository for word segmentation in sanskrit using energy based models.&lt;/p&gt;

&lt;p&gt;&amp;nbsp;&lt;/p&gt;

&lt;p&gt;# Word Segmentation in Sanskrit Using Energy Based Models&lt;br&gt;
&lt;br&gt;
&amp;nbsp;&lt;br&gt;
## Getting Started&lt;br&gt;
&amp;nbsp;&lt;br&gt;
Please download the 2 compressed files &amp;#39;dir.zip&amp;#39; and &amp;#39;wordsegmentation.rar&amp;#39; to your working directory and extract them into folders named &amp;#39;dir&amp;#39; and &amp;#39;wordsegmentation&amp;#39; respectively.&lt;br&gt;
&amp;nbsp;&lt;br&gt;
Your working directory should be as follows&lt;br&gt;
* Working Directory&lt;br&gt;
&amp;nbsp; * wordsegmentation&lt;br&gt;
&amp;nbsp;&amp;nbsp;&amp;nbsp; * skt_dcs_DS.bz2_4K_bigram_mir_10K&lt;br&gt;
&amp;nbsp;&amp;nbsp;&amp;nbsp; * skt_dcs_DS.bz2_4K_bigram_mir_heldout&lt;br&gt;
&amp;nbsp; * dir&lt;br&gt;
&amp;nbsp;&lt;br&gt;
## Prerequisites&lt;br&gt;
* Python3&lt;br&gt;
&amp;nbsp; * scipy&lt;br&gt;
&amp;nbsp; * numpy&lt;br&gt;
&amp;nbsp; * csv&lt;br&gt;
&amp;nbsp; * pickle&lt;br&gt;
&amp;nbsp; * multiprocessing&lt;br&gt;
&amp;nbsp; * bz2&lt;br&gt;
## Instructions for Training&lt;br&gt;
Change your current directory to &amp;#39;dir&amp;#39;&lt;br&gt;
&amp;nbsp;&lt;br&gt;
Run the file Train_clique.py by using the following command&lt;br&gt;
&amp;nbsp;&lt;br&gt;
* python Train_clique.py&lt;br&gt;
&amp;nbsp;&lt;br&gt;
To train on different input features like BM2,BM3,BR2,BR3,PM2,PM3,PR,PR3 please modify the bz2_input_folder value in the main function before beginning the training.&lt;br&gt;
&amp;nbsp;&lt;br&gt;
Feature&amp;nbsp; | bz2_input_folder&lt;br&gt;
------------- | -------------&lt;br&gt;
BM2 | wordsegmentation/skt_dcs_DS.bz2_4K_bigram_mir_10K/&lt;br&gt;
BM3 | wordsegmentation/skt_dcs_DS.bz2_1L_bigram_mir_10K&lt;br&gt;
BR2 | wordsegmentation/skt_dcs_DS.bz2_4K_bigram_rfe_10K/&lt;br&gt;
BR3 | wordsegmentation/skt_dcs_DS.bz2_1L_bigram_rfe_10K/&lt;br&gt;
PM2 | wordsegmentation/skt_dcs_DS.bz2_4K_pmi_mir_10K/&lt;br&gt;
PM3 | wordsegmentation/skt_dcs_DS.bz2_1L_pmi_mir_10K2/&lt;br&gt;
PR2 | wordsegmentation/skt_dcs_DS.bz2_4K_pmi_rfe_10K/&lt;br&gt;
PR3 | wordsegmentation/skt_dcs_DS.bz2_1L_pmi_rfe_10K/&lt;br&gt;
&amp;nbsp;&lt;br&gt;
## Instructions for Testing&lt;br&gt;
&amp;nbsp;&lt;br&gt;
After training, please modify the &amp;#39;modelList&amp;#39; dictionary&amp;nbsp; in &amp;#39;test_clique.py&amp;#39; with the name of the neural network that has been saved during training. While testing for a feature, please provide the name of the neural net which was trained for the same feature.&lt;br&gt;
&amp;nbsp;&lt;br&gt;
We only provide the trained model for the feature BM2 which was our best performing feature. If the name of the neural net is not changed, then the testing will be performed on the pre-trained model for BM2 provided in outputs/train_t7978754709018&lt;br&gt;
&amp;nbsp;&lt;br&gt;
To test with a particular feature vector use the tag of the feature while execution&lt;br&gt;
&amp;nbsp;&lt;br&gt;
* python test_clique.py -t &amp;lt;tag&amp;gt;&lt;br&gt;
&amp;nbsp;&lt;br&gt;
For example: &amp;nbsp;&lt;br&gt;
&amp;nbsp; * python test_clique.py -t BM2&lt;br&gt;
&amp;nbsp;&lt;br&gt;
After finishing the testing please run the following command to see the precision and recall values for both the word and word++ prediction tasks&lt;br&gt;
&amp;nbsp;&lt;br&gt;
* python evaluate.py &amp;lt;tag&amp;gt;&lt;br&gt;
&amp;nbsp;&lt;br&gt;
For example: &amp;nbsp;&lt;br&gt;
&amp;nbsp; * python evaluate.py BM2&lt;/p&gt;</description>
  </descriptions>
</resource>
653
531
views
downloads
All versions This version
Views 653654
Downloads 531531
Data volume 13.0 TB13.0 TB
Unique views 609610
Unique downloads 248248

Share

Cite as