edu.northwestern.at.morphadorner.tools.createlexicon

## Class CreateLexicon

- java.lang.Object
- - edu.northwestern.at.morphadorner.tools.createlexicon.CreateLexicon

-     public class CreateLexicon
      extends java.lang.Object

  Generate lexicons from training data.

  `java -Xmx512m edu.northwestern.at.morphadorner.tools.createlexicon.CreateLexicon trainingdata outputwordlexicon outputsuffixlexicon maxsuffixlength maxsuffixcount`

  - trainingdata specifies the name of the file containing the part of speech training data from which the word lexicon and suffix lexicon are built. The word lexicon contains each spelling (and standard spellings if provided), the count for each spelling, the parts of speech for each spelling, the counts for each part of speech for each spelling, and the lemma for each part of speech for each spelling (if provided). The suffix lexicon contains a list of suffixes, their counts, and the parts of speech associated with each suffix and the count of each part of speech. Lemmata are stored as a "*' in the suffix lexicon since there are no lemmata for suffixes.

    The training data resides in a utf-8 text file. Each line contains one tab-separated spelling along with its part of speech tag and optionally its lemma and standard spelling in the form:

    `spellingpart-of-speech-taglemmastandardspelling`

    You must specify a spelling and a part of speech tag. The lemma and standard spelling are optional. If you wish to specify a standard spelling without specifying a lemma, enter the lemma as "*".

    Blank lines are used to separate sentences. While the blank lines are not needed for creating the lexicon, they are needed for creating probability transition matrices and for part of speech tagging.

    The lexicon is built using both the spelling and the standard spelling (when provided). The lemma is also stored when present.

  - outputwordlexicon specifies the name of the output file to receive the word lexicon.

  - outputsuffixlexicon specifies the name of the output file to receive tthe suffix lexicon.

  - maxsuffixlength specifies the maximum length suffix generated for the suffix lexicon. The default is 6.

  - maxsuffixcount specifies the maximum number of times a spelling can appear in order for its suffix to be added to the suffix lexicon. The default is to include all words regardless of count.

    For some applications you may want to restrict the suffix lexicon to contain suffixes only for infrequently occurring words. Values of 10 (only include spellings which appear 10 or less times in the training data) or 1 (only include spellings which appear once in the training data) are popular choices.

- - ### Field Summary

    Fields
    Modifier and Type | Field and Description
    --- | ---
    `protected static int` | `maxSuffixCount` Only use words less than maxSuffixCount to generate suffix lexicon.
    `protected static int` | `maxSuffixLength` Maximum and minimum length suffixes to generated.
    `protected static int` | `minSuffixLength`
    `protected static java.lang.String` | `suffixLexiconFileName` Output suffix lexicon file name.
    `protected static java.lang.String` | `trainingDataFileName` Training data file name.
    `protected static java.lang.String` | `wordLexiconFileName` Output word lexicon file name.

  - ### Constructor Summary

    Constructors
    Constructor and Description
    `CreateLexicon()`

  - ### Method Summary

    Methods
    Modifier and Type | Method and Description
    --- | ---
    `protected static void` | `help()` Display brief help.
    `protected static boolean` | `initialize(java.lang.String[] args)` Initialize.
    `static void` | `main(java.lang.String[] args)` Main program.

    - ### Methods inherited from class java.lang.Object

      `clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- - ### Field Detail

    - #### trainingDataFileName

          protected static java.lang.String trainingDataFileName

      Training data file name.

    - #### wordLexiconFileName

          protected static java.lang.String wordLexiconFileName

      Output word lexicon file name.

    - #### suffixLexiconFileName

          protected static java.lang.String suffixLexiconFileName

      Output suffix lexicon file name.

    - #### maxSuffixCount

          protected static int maxSuffixCount

      Only use words less than maxSuffixCount to generate suffix lexicon.

      The default is to use all words regardless of word count.

    - #### maxSuffixLength

          protected static int maxSuffixLength

      Maximum and minimum length suffixes to generated.

    - #### minSuffixLength

          protected static int minSuffixLength

  - ### Constructor Detail

    - #### CreateLexicon

          public CreateLexicon()

  - ### Method Detail

    - #### help

          protected static void help()

      Display brief help.

    - #### initialize

          protected static boolean initialize(java.lang.String[] args)

      Initialize.
      Parameters:
        `args` - Command line arguments.

    - #### main

          public static void main(java.lang.String[] args)

      Main program.
      Parameters:
        `args` - Command line arguments.