edu.northwestern.at.morphadorner.corpuslinguistics.lexicon

## Class AbstractLexicon

- java.lang.Object
- - edu.northwestern.at.utils.IsCloseableObject
  - - edu.northwestern.at.morphadorner.corpuslinguistics.lexicon.AbstractLexicon

- All Implemented Interfaces:
    Lexicon, UsesLogger
  Direct Known Subclasses:
    BaseLexicon, PennTreebankSuffixLexicon, PennTreebankWordLexicon

      public abstract class AbstractLexicon
      extends IsCloseableObject
      implements Lexicon, UsesLogger

  AbstractLexicon: stores spellings and their possible parts of speech.

  Each line in the main lexicon file takes the following form:

  > `spelling countspelling pos1 countpos1 pos2 countpos2 ...`

  where spelling is the spelling for a word, countspelling is the number of times the spelling appeared in the training data, pos1 is the tag corresponding to the most commonly occurring part of speech for this spelling, countpos1 is the number of times the pos1 tag appeared, and pos2, countpos2, etc. are the other possible parts of speech and their counts.

  The raw counts are stored rather than probabilities so that new training data can be used to update the lexicon easily, and so that individual part of speech taggers can apply different methods of count smoothing.

- - ### Field Summary

    Fields
    Modifier and Type | Field and Description
    --- | ---
    `protected java.util.Map<java.lang.String,MutableInteger>` | `categoryCountsMap` Map from part of speech tags to their frequency in the lexicon.
    `protected java.util.Map<java.lang.String,LexiconEntry>` | `lexiconMap` Map in which to store lexicon entries.
    `protected Logger` | `logger` Logger used for output.
    `protected int` | `longestEntryLength` Length (in characters) of the longest entry in the lexicon.
    `protected PartOfSpeechTags` | `partOfSpeechTags` Part of Speech tag set used by lexicon.
    `protected int` | `shortestEntryLength` Length (in characters) of the shortest entry in the lexicon.
    `protected java.util.Map<java.lang.String,MutableInteger>` | `uniqueEntryCountForCategoryMap` Map from part of speech tags to frequency of unique word entries in the lexicon with each tag.

  - ### Constructor Summary

    Constructors
    Constructor and Description
    `AbstractLexicon()` Create an empty lexicon.

  - ### Method Summary

    Methods
    Modifier and Type | Method and Description
    --- | ---
    `protected boolean` | `checkCategoriesList()` Check that all the tags in the lexicon appear in the designated part of speech tags list.
    `protected void` | `computeUniqueEntryCountsForCategories()` Compute number of lexicon entries for each category.
    `boolean` | `containsEntry(java.lang.String entry)` Checks if lexicon contains an entry.
    `java.lang.String[]` | `getCategories()` Get the categories, sorted in ascending order.
    `java.util.Set<java.lang.String>` | `getCategoriesForEntry(java.util.List<java.lang.String> sentence, int entryIndex)` Get categories for an entry in a sentence.
    `java.util.Set<java.lang.String>` | `getCategoriesForEntry(java.lang.String entry)` Get categories for an entry in the lexicon.
    `java.util.Set<java.lang.String>` | `getCategoriesForEntry(java.lang.String entry, boolean isFirstEntry)` Get categories for an entry.
    `int` | `getCategoryCount(java.lang.String category)` Get category count.
    `int` | `getCategoryCount(java.lang.String entry, java.lang.String category)` Get count for an entry in a specific category.
    `java.util.Map<java.lang.String,MutableInteger>` | `getCategoryCounts()` Get category counts.
    `java.util.Map<java.lang.String,MutableInteger>` | `getCategoryCountsForEntry(java.lang.String entry)` Get category counts for an entry.
    `java.lang.String[]` | `getEntries()` Get the entries, sorted in ascending order.
    `int` | `getEntryCount(java.lang.String entry)` Get total count for an entry.
    `java.lang.String` | `getLargestCategory(java.lang.String entry)` Get category with largest count for an entry.
    `java.lang.String` | `getLemma(java.lang.String entry)` Get lemma for an entry.
    `java.lang.String` | `getLemma(java.lang.String entry, java.lang.String category)` Get lemma for an entry in a specific category.
    `java.lang.String[]` | `getLemmata(java.lang.String entry)` Get all lemmata for an entry.
    `LexiconEntry` | `getLexiconEntry(java.lang.String entry)` Get a lexicon entry.
    `int` | `getLexiconSize()` Get number of entries in Lexicon.
    `Logger` | `getLogger()` Get the logger.
    `int` | `getLongestEntryLength()` Get the longest entry length in the lexicon.
    `int` | `getNumberOfCategories()` Get number of categories.
    `int` | `getNumberOfCategoriesForEntry(java.lang.String entry)` Get number of categories for an entry.
    `PartOfSpeechTags` | `getPartOfSpeechTags()` Get the part of speech tags list used by the lexicon.
    `int` | `getShortestEntryLength()` Get the shortest entry length in the lexicon.
    `int` | `getUniqueEntryCountForCategory(java.lang.String category)` Get unique entry count for a category.
    `protected void` | `incrementUniqueEntryCountForCategory(java.lang.String category)` Increment number of unique entries for a category.
    `void` | `loadLexicon(java.net.URL lexiconURL, boolean compressed, java.lang.String encoding)` Load entries into a lexicon.
    `void` | `loadLexicon(java.net.URL lexiconURL, java.lang.String encoding)` Load entries into a lexicon.
    `void` | `removeEntry(java.lang.String entry)` Remove entry.
    `void` | `removeEntryCategory(java.lang.String entry, java.lang.String category)` Remove given category for an entry.
    `void` | `saveLexiconToTextFile(java.lang.String lexiconFileName, java.lang.String encoding)` Save lexicon to a file.
    `LexiconEntry` | `setLexiconEntry(java.lang.String entry, LexiconEntry entryData)` Set a lexicon entry.
    `void` | `setLogger(Logger logger)` Set the logger.
    `boolean` | `setPartOfSpeechTags(PartOfSpeechTags partOfSpeechTags)` Set the part of speech tags list used by the lexicon.
    `protected void` | `updateCategoryCount(java.lang.String category, int count)` Add or update category counts map.
    `void` | `updateEntryCount(java.lang.String entry, java.lang.String category, java.lang.String lemma, int entryCount)` Update entry count in lexicon for a given category.

    - ### Methods inherited from class edu.northwestern.at.utils.IsCloseableObject

      `close`

    - ### Methods inherited from class java.lang.Object

      `clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- - ### Field Detail

    - #### lexiconMap

          protected java.util.Map<java.lang.String,LexiconEntry> lexiconMap

      Map in which to store lexicon entries.

      An entry (e.g., word spelling) is the key, and a LexiconEntry is the value.

    - #### categoryCountsMap

          protected java.util.Map<java.lang.String,MutableInteger> categoryCountsMap

      Map from part of speech tags to their frequency in the lexicon.

    - #### uniqueEntryCountForCategoryMap

          protected java.util.Map<java.lang.String,MutableInteger> uniqueEntryCountForCategoryMap

      Map from part of speech tags to frequency of unique word entries in the lexicon with each tag.

    - #### longestEntryLength

          protected int longestEntryLength

      Length (in characters) of the longest entry in the lexicon.

    - #### shortestEntryLength

          protected int shortestEntryLength

      Length (in characters) of the shortest entry in the lexicon.

    - #### partOfSpeechTags

          protected PartOfSpeechTags partOfSpeechTags

      Part of Speech tag set used by lexicon.

      Note: all tags in the lexicon must appear in this list!

    - #### logger

          protected Logger logger

      Logger used for output.

  - ### Constructor Detail

    - #### AbstractLexicon

          public AbstractLexicon()

      Create an empty lexicon.

  - ### Method Detail

    - #### getLogger

          public Logger getLogger()

      Get the logger.
      Specified by:
        `getLogger` in interface `UsesLogger`
      Returns:
        The logger.

    - #### setLogger

          public void setLogger(Logger logger)

      Set the logger.
      Specified by:
        `setLogger` in interface `UsesLogger`
      Parameters:
        `logger` - The logger.

    - #### updateCategoryCount

          protected void updateCategoryCount(java.lang.String category,
                                 int count)

      Add or update category counts map.
      Parameters:
        `category` - Category for which to add/update count.
        `count` - Category count to add to entry. May be negative.

    - #### incrementUniqueEntryCountForCategory

          protected void incrementUniqueEntryCountForCategory(java.lang.String category)

      Increment number of unique entries for a category.
      Parameters:
        `category` - Category for which to increment count.

    - #### updateEntryCount

          public void updateEntryCount(java.lang.String entry,
                              java.lang.String category,
                              java.lang.String lemma,
                              int entryCount)

      Update entry count in lexicon for a given category.
      Specified by:
        `updateEntryCount` in interface `Lexicon`
      Parameters:
        `entry` - The entry.
        `category` - The category.
        `lemma` - The lemma.
        `entryCount` - The entry count to add to the current count. Must be positive.

    - #### removeEntryCategory

          public void removeEntryCategory(java.lang.String entry,
                                 java.lang.String category)

      Remove given category for an entry.
      Specified by:
        `removeEntryCategory` in interface `Lexicon`
      Parameters:
        `entry` - The entry.
        `category` - The category to remove.

        If the entry has no remaining categories, the entry is removed from the lexicon.

    - #### removeEntry

          public void removeEntry(java.lang.String entry)

      Remove entry.
      Specified by:
        `removeEntry` in interface `Lexicon`
      Parameters:
        `entry` - The entry to remove.

    - #### loadLexicon

          public void loadLexicon(java.net.URL lexiconURL,
                         java.lang.String encoding)
                           throws java.io.IOException

      Load entries into a lexicon.
      Specified by:
        `loadLexicon` in interface `Lexicon`
      Parameters:
        `lexiconURL` - URL for the file containing the lexicon.
        `encoding` - Character encoding of lexicon file text.
      Throws:
        `java.io.IOException`

    - #### loadLexicon

          public void loadLexicon(java.net.URL lexiconURL,
                         boolean compressed,
                         java.lang.String encoding)
                           throws java.io.IOException

      Load entries into a lexicon.
      Specified by:
        `loadLexicon` in interface `Lexicon`
      Parameters:
        `lexiconURL` - URL for the file containing the lexicon.
        `compressed` - true if lexicon is gzip compressed.
        `encoding` - Character encoding of lexicon file text.
      Throws:
        `java.io.IOException`

    - #### computeUniqueEntryCountsForCategories

          protected void computeUniqueEntryCountsForCategories()

      Compute number of lexicon entries for each category.

    - #### getLexiconSize

          public int getLexiconSize()

      Get number of entries in Lexicon.
      Specified by:
        `getLexiconSize` in interface `Lexicon`
      Returns:
        Number of entries in Lexicon.

        Returns number of fixed entries

    - #### getEntries

          public java.lang.String[] getEntries()

      Get the entries, sorted in ascending order.
      Specified by:
        `getEntries` in interface `Lexicon`
      Returns:
        The sorted entry strings as an array of string.

    - #### getCategories

          public java.lang.String[] getCategories()

      Get the categories, sorted in ascending order.
      Specified by:
        `getCategories` in interface `Lexicon`
      Returns:
        The sorted category strings as an array of string.

    - #### containsEntry

          public boolean containsEntry(java.lang.String entry)

      Checks if lexicon contains an entry.
      Specified by:
        `containsEntry` in interface `Lexicon`
      Parameters:
        `entry` - Entry to look up.
      Returns:
        true if lexicon contains entry. Only an exact match is considered.

    - #### getLexiconEntry

          public LexiconEntry getLexiconEntry(java.lang.String entry)

      Get a lexicon entry.
      Specified by:
        `getLexiconEntry` in interface `Lexicon`
      Parameters:
        `entry` - Entry for which to get lexicon information.
      Returns:
        LexiconEntry for entry, or null if not found.

        Note: this does NOT call the part of speech guesser.

    - #### setLexiconEntry

          public LexiconEntry setLexiconEntry(java.lang.String entry,
                                     LexiconEntry entryData)

      Set a lexicon entry.
      Specified by:
        `setLexiconEntry` in interface `Lexicon`
      Parameters:
        `entry` - Entry for which to get lexicon information.
        `entryData` - The lexicon entry data.
      Returns:
        Previous lexicon data for entry, if any.

    - #### getCategoriesForEntry

          public java.util.Set<java.lang.String> getCategoriesForEntry(java.lang.String entry)

      Get categories for an entry in the lexicon.
      Specified by:
        `getCategoriesForEntry` in interface `Lexicon`
      Parameters:
        `entry` - Entry to look up.
      Returns:
        Set of categories. Null if entry not found in lexicon.

    - #### getCategoriesForEntry

          public java.util.Set<java.lang.String> getCategoriesForEntry(java.util.List<java.lang.String> sentence,
                                                              int entryIndex)

      Get categories for an entry in a sentence.
      Specified by:
        `getCategoriesForEntry` in interface `Lexicon`
      Parameters:
        `sentence` - List of entries in sentence.
        `entryIndex` - Index within sentence (0-based) of entry.
      Returns:
        Set of categories. Null if entry not found in lexicon.

    - #### getCategoriesForEntry

          public java.util.Set<java.lang.String> getCategoriesForEntry(java.lang.String entry,
                                                              boolean isFirstEntry)

      Get categories for an entry.
      Specified by:
        `getCategoriesForEntry` in interface `Lexicon`
      Parameters:
        `entry` - Entry to look up.
        `isFirstEntry` - True if entry is first in sentence.
      Returns:
        Set of categories. Null if entry not found in lexicon.

    - #### getNumberOfCategoriesForEntry

          public int getNumberOfCategoriesForEntry(java.lang.String entry)

      Get number of categories for an entry.
      Specified by:
        `getNumberOfCategoriesForEntry` in interface `Lexicon`
      Parameters:
        `entry` - Entry for which to find number of categories.
      Returns:
        Number of categories for entry.

    - #### getLargestCategory

          public java.lang.String getLargestCategory(java.lang.String entry)

      Get category with largest count for an entry.
      Specified by:
        `getLargestCategory` in interface `Lexicon`
      Parameters:
        `entry` - Entry to look up.
      Returns:
        Category with largest count. Null if entry not found in lexicon.

    - #### getCategoryCount

          public int getCategoryCount(java.lang.String category)

      Get category count.
      Specified by:
        `getCategoryCount` in interface `Lexicon`
      Parameters:
        `category` - Get number of times category appears in lexicon.
      Returns:
        Category count.

    - #### getUniqueEntryCountForCategory

          public int getUniqueEntryCountForCategory(java.lang.String category)

      Get unique entry count for a category.
      Parameters:
        `category` - Category.
      Returns:
        Count of unique entries with this category.

    - #### getCategoryCount

          public int getCategoryCount(java.lang.String entry,
                             java.lang.String category)

      Get count for an entry in a specific category.
      Specified by:
        `getCategoryCount` in interface `Lexicon`
      Parameters:
        `entry` - Entry to look up.
        `category` - Category for which to retrieve count.
      Returns:
        Number of occurrences of entry in category.

    - #### getLemma

          public java.lang.String getLemma(java.lang.String entry)

      Get lemma for an entry.
      Specified by:
        `getLemma` in interface `Lexicon`
      Parameters:
        `entry` - Entry to look up.
      Returns:
        Lemma form of entry. A "*' is returned if the lemma cannot be found.

        Some spellings may have multiple lemmata depending upon the part of speech. This method returns the lemma associated with the most frequently occurring part of speech.

    - #### getLemmata

          public java.lang.String[] getLemmata(java.lang.String entry)

      Get all lemmata for an entry.
      Specified by:
        `getLemmata` in interface `Lexicon`
      Parameters:
        `entry` - Entry to look up.
      Returns:
        Lemmata forms of entry.

    - #### getLemma

          public java.lang.String getLemma(java.lang.String entry,
                                  java.lang.String category)

      Get lemma for an entry in a specific category.
      Specified by:
        `getLemma` in interface `Lexicon`
      Parameters:
        `entry` - Entry to look up.
        `category` - Category for which to retrieve lemma.
      Returns:
        Lemma form of entry. An "*' is returned if the lemma cannot be found.

    - #### getCategoryCounts

          public java.util.Map<java.lang.String,MutableInteger> getCategoryCounts()

      Get category counts.
      Specified by:
        `getCategoryCounts` in interface `Lexicon`
      Returns:
        Category counts map.

    - #### getNumberOfCategories

          public int getNumberOfCategories()

      Get number of categories.
      Specified by:
        `getNumberOfCategories` in interface `Lexicon`
      Returns:
        Number of categories.

    - #### getCategoryCountsForEntry

          public java.util.Map<java.lang.String,MutableInteger> getCategoryCountsForEntry(java.lang.String entry)

      Get category counts for an entry.
      Specified by:
        `getCategoryCountsForEntry` in interface `Lexicon`
      Parameters:
        `entry` - Entry to look up.
      Returns:
        Map of counts for each category. String keys are tags, MutableInteger counts are values. Null if entry not found in lexicon.

    - #### getEntryCount

          public int getEntryCount(java.lang.String entry)

      Get total count for an entry.
      Specified by:
        `getEntryCount` in interface `Lexicon`
      Parameters:
        `entry` - Entry to look up.
      Returns:
        Count of occurrences of entry.

    - #### saveLexiconToTextFile

          public void saveLexiconToTextFile(java.lang.String lexiconFileName,
                                   java.lang.String encoding)
                                     throws java.io.IOException

      Save lexicon to a file.
      Specified by:
        `saveLexiconToTextFile` in interface `Lexicon`
      Parameters:
        `lexiconFileName` - File containing the lexicon.
        `encoding` - Character encoding of lexicon file text.
      Throws:
        `java.io.IOException`

    - #### getLongestEntryLength

          public int getLongestEntryLength()

      Get the longest entry length in the lexicon.
      Specified by:
        `getLongestEntryLength` in interface `Lexicon`
      Returns:
        The longest entry length in the lexicon.

    - #### getShortestEntryLength

          public int getShortestEntryLength()

      Get the shortest entry length in the lexicon.
      Specified by:
        `getShortestEntryLength` in interface `Lexicon`
      Returns:
        The shortest entry length in the lexicon.

    - #### checkCategoriesList

          protected boolean checkCategoriesList()

      Check that all the tags in the lexicon appear in the designated part of speech tags list.
      Returns:
        true if all tags used in lexicon appear in designated part of speech list.

    - #### getPartOfSpeechTags

          public PartOfSpeechTags getPartOfSpeechTags()

      Get the part of speech tags list used by the lexicon.
      Specified by:
        `getPartOfSpeechTags` in interface `Lexicon`
      Returns:
        Part of speech tags list.

    - #### setPartOfSpeechTags

          public boolean setPartOfSpeechTags(PartOfSpeechTags partOfSpeechTags)

      Set the part of speech tags list used by the lexicon.
      Specified by:
        `setPartOfSpeechTags` in interface `Lexicon`
      Parameters:
        `partOfSpeechTags` - Part of speech tags list.
      Returns:
        true if all categories in lexicon appear in the part of speech tags list.

        For the check to work, the part of speech tags list should be set after the lexicon is loaded.