August 23, 2018 Software Open Access

Amrith Krishna; Bishal Santra; Sasi Prasanth Bandaru; Gaurav Sahu; Vishnu Dutt Sharma; Pavankumar Satuluri; Pawan Goyal

MARC21 XML Export

<?xml version='1.0' encoding='UTF-8'?>
<record xmlns="http://www.loc.gov/MARC21/slim">
  <leader>00000nmm##2200000uu#4500</leader>
  <controlfield tag="005">20200125072505.0</controlfield>
  <controlfield tag="001">1035413</controlfield>
  <datafield tag="711" ind1=" " ind2=" ">
    <subfield code="d">October 31–November 4 @018</subfield>
    <subfield code="g">EMNLP`</subfield>
    <subfield code="a">Conference on Empirical Methods in Natural Language Processing</subfield>
    <subfield code="c">Brussels, Belgium</subfield>
  </datafield>
  <datafield tag="700" ind1=" " ind2=" ">
    <subfield code="u">IIT Kharagpur</subfield>
    <subfield code="a">Bishal Santra</subfield>
  </datafield>
  <datafield tag="700" ind1=" " ind2=" ">
    <subfield code="u">IIT Kharagpur</subfield>
    <subfield code="a">Sasi Prasanth Bandaru</subfield>
  </datafield>
  <datafield tag="700" ind1=" " ind2=" ">
    <subfield code="u">IIT Kharagpur</subfield>
    <subfield code="a">Gaurav Sahu</subfield>
  </datafield>
  <datafield tag="700" ind1=" " ind2=" ">
    <subfield code="u">American Express</subfield>
    <subfield code="a">Vishnu Dutt Sharma</subfield>
  </datafield>
  <datafield tag="700" ind1=" " ind2=" ">
    <subfield code="u">Chinmya Visvavidyapeeth</subfield>
    <subfield code="a">Pavankumar Satuluri</subfield>
  </datafield>
  <datafield tag="700" ind1=" " ind2=" ">
    <subfield code="u">IIT Kharagpur</subfield>
    <subfield code="a">Pawan Goyal</subfield>
  </datafield>
  <datafield tag="856" ind1="4" ind2=" ">
    <subfield code="s">453229783</subfield>
    <subfield code="z">md5:016462cbd311404a6c9fb9af950d38a5</subfield>
    <subfield code="u">https://zenodo.org/record/1035413/files/dir.zip</subfield>
  </datafield>
  <datafield tag="856" ind1="4" ind2=" ">
    <subfield code="s">2418</subfield>
    <subfield code="z">md5:c0163b57ec0ab0013603d017556e2f2b</subfield>
    <subfield code="u">https://zenodo.org/record/1035413/files/README.md</subfield>
  </datafield>
  <datafield tag="856" ind1="4" ind2=" ">
    <subfield code="s">41733267455</subfield>
    <subfield code="z">md5:6339b68e76df5aab37d2850fccf68c98</subfield>
    <subfield code="u">https://zenodo.org/record/1035413/files/wordsegmentation.rar</subfield>
  </datafield>
  <datafield tag="542" ind1=" " ind2=" ">
    <subfield code="l">open</subfield>
  </datafield>
  <datafield tag="856" ind1="4" ind2=" ">
    <subfield code="y">Conference website</subfield>
    <subfield code="u">http://emnlp2018.org/</subfield>
  </datafield>
  <datafield tag="260" ind1=" " ind2=" ">
    <subfield code="c">2018-08-23</subfield>
  </datafield>
  <datafield tag="909" ind1="C" ind2="O">
    <subfield code="p">software</subfield>
    <subfield code="p">user-cnerg</subfield>
    <subfield code="o">oai:zenodo.org:1035413</subfield>
  </datafield>
  <datafield tag="100" ind1=" " ind2=" ">
    <subfield code="u">IIT Kharagpur</subfield>
    <subfield code="a">Amrith Krishna</subfield>
  </datafield>
  <datafield tag="245" ind1=" " ind2=" ">
    <subfield code="a">Word Segmentation in Sanskrit Using Energy Based Models</subfield>
  </datafield>
  <datafield tag="980" ind1=" " ind2=" ">
    <subfield code="a">user-cnerg</subfield>
  </datafield>
  <datafield tag="540" ind1=" " ind2=" ">
    <subfield code="u">https://creativecommons.org/licenses/by/4.0/legalcode</subfield>
    <subfield code="a">Creative Commons Attribution 4.0 International</subfield>
  </datafield>
  <datafield tag="650" ind1="1" ind2="7">
    <subfield code="a">cc-by</subfield>
    <subfield code="2">opendefinition.org</subfield>
  </datafield>
  <datafield tag="520" ind1=" " ind2=" ">
    <subfield code="a">&lt;p&gt;This is the repository for word segmentation in sanskrit using energy based models.&lt;/p&gt;

&lt;p&gt;&amp;nbsp;&lt;/p&gt;

&lt;p&gt;# Word Segmentation in Sanskrit Using Energy Based Models&lt;br&gt;
&lt;br&gt;
&amp;nbsp;&lt;br&gt;
## Getting Started&lt;br&gt;
&amp;nbsp;&lt;br&gt;
Please download the 2 compressed files &amp;#39;dir.zip&amp;#39; and &amp;#39;wordsegmentation.rar&amp;#39; to your working directory and extract them into folders named &amp;#39;dir&amp;#39; and &amp;#39;wordsegmentation&amp;#39; respectively.&lt;br&gt;
&amp;nbsp;&lt;br&gt;
Your working directory should be as follows&lt;br&gt;
* Working Directory&lt;br&gt;
&amp;nbsp; * wordsegmentation&lt;br&gt;
&amp;nbsp;&amp;nbsp;&amp;nbsp; * skt_dcs_DS.bz2_4K_bigram_mir_10K&lt;br&gt;
&amp;nbsp;&amp;nbsp;&amp;nbsp; * skt_dcs_DS.bz2_4K_bigram_mir_heldout&lt;br&gt;
&amp;nbsp; * dir&lt;br&gt;
&amp;nbsp;&lt;br&gt;
## Prerequisites&lt;br&gt;
* Python3&lt;br&gt;
&amp;nbsp; * scipy&lt;br&gt;
&amp;nbsp; * numpy&lt;br&gt;
&amp;nbsp; * csv&lt;br&gt;
&amp;nbsp; * pickle&lt;br&gt;
&amp;nbsp; * multiprocessing&lt;br&gt;
&amp;nbsp; * bz2&lt;br&gt;
## Instructions for Training&lt;br&gt;
Change your current directory to &amp;#39;dir&amp;#39;&lt;br&gt;
&amp;nbsp;&lt;br&gt;
Run the file Train_clique.py by using the following command&lt;br&gt;
&amp;nbsp;&lt;br&gt;
* python Train_clique.py&lt;br&gt;
&amp;nbsp;&lt;br&gt;
To train on different input features like BM2,BM3,BR2,BR3,PM2,PM3,PR,PR3 please modify the bz2_input_folder value in the main function before beginning the training.&lt;br&gt;
&amp;nbsp;&lt;br&gt;
Feature&amp;nbsp; | bz2_input_folder&lt;br&gt;
------------- | -------------&lt;br&gt;
BM2 | wordsegmentation/skt_dcs_DS.bz2_4K_bigram_mir_10K/&lt;br&gt;
BM3 | wordsegmentation/skt_dcs_DS.bz2_1L_bigram_mir_10K&lt;br&gt;
BR2 | wordsegmentation/skt_dcs_DS.bz2_4K_bigram_rfe_10K/&lt;br&gt;
BR3 | wordsegmentation/skt_dcs_DS.bz2_1L_bigram_rfe_10K/&lt;br&gt;
PM2 | wordsegmentation/skt_dcs_DS.bz2_4K_pmi_mir_10K/&lt;br&gt;
PM3 | wordsegmentation/skt_dcs_DS.bz2_1L_pmi_mir_10K2/&lt;br&gt;
PR2 | wordsegmentation/skt_dcs_DS.bz2_4K_pmi_rfe_10K/&lt;br&gt;
PR3 | wordsegmentation/skt_dcs_DS.bz2_1L_pmi_rfe_10K/&lt;br&gt;
&amp;nbsp;&lt;br&gt;
## Instructions for Testing&lt;br&gt;
&amp;nbsp;&lt;br&gt;
After training, please modify the &amp;#39;modelList&amp;#39; dictionary&amp;nbsp; in &amp;#39;test_clique.py&amp;#39; with the name of the neural network that has been saved during training. While testing for a feature, please provide the name of the neural net which was trained for the same feature.&lt;br&gt;
&amp;nbsp;&lt;br&gt;
We only provide the trained model for the feature BM2 which was our best performing feature. If the name of the neural net is not changed, then the testing will be performed on the pre-trained model for BM2 provided in outputs/train_t7978754709018&lt;br&gt;
&amp;nbsp;&lt;br&gt;
To test with a particular feature vector use the tag of the feature while execution&lt;br&gt;
&amp;nbsp;&lt;br&gt;
* python test_clique.py -t &amp;lt;tag&amp;gt;&lt;br&gt;
&amp;nbsp;&lt;br&gt;
For example: &amp;nbsp;&lt;br&gt;
&amp;nbsp; * python test_clique.py -t BM2&lt;br&gt;
&amp;nbsp;&lt;br&gt;
After finishing the testing please run the following command to see the precision and recall values for both the word and word++ prediction tasks&lt;br&gt;
&amp;nbsp;&lt;br&gt;
* python evaluate.py &amp;lt;tag&amp;gt;&lt;br&gt;
&amp;nbsp;&lt;br&gt;
For example: &amp;nbsp;&lt;br&gt;
&amp;nbsp; * python evaluate.py BM2&lt;/p&gt;</subfield>
  </datafield>
  <datafield tag="773" ind1=" " ind2=" ">
    <subfield code="n">doi</subfield>
    <subfield code="i">isVersionOf</subfield>
    <subfield code="a">10.5281/zenodo.1035412</subfield>
  </datafield>
  <datafield tag="024" ind1=" " ind2=" ">
    <subfield code="a">10.5281/zenodo.1035413</subfield>
    <subfield code="2">doi</subfield>
  </datafield>
  <datafield tag="980" ind1=" " ind2=" ">
    <subfield code="a">software</subfield>
  </datafield>
</record>