edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer

## Class AbstractWordTokenizer

- java.lang.Object
- - edu.northwestern.at.utils.IsCloseableObject
  - - edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer.AbstractWordTokenizer

- All Implemented Interfaces:
    WordTokenizer, IsCloseable, UsesLogger
  Direct Known Subclasses:
    ContractionTokenizer, DefaultWordTokenizer, ICU4JBreakIteratorWordTokenizer, NoopWordTokenizer, PennTreebankTokenizer, WhitespaceWordTokenizer

      public abstract class AbstractWordTokenizer
      extends IsCloseableObject
      implements WordTokenizer, IsCloseable, UsesLogger

  Base class for deriving word tokenizers.

- - ### Field Summary

    Fields
    Modifier and Type | Field and Description
    --- | ---
    `protected Abbreviations` | `abbreviations` Abbreviations.
    `protected AposTokens` | `aposTokens` Apostrophe tokens.
    `protected boolean` | `apostropheCanBeQuote` True if apostrophes can be single quotes.
    `protected boolean` | `coalesceAsterisks` True to coalesce adjacent asterisks.
    `protected boolean` | `coalesceHyphens` True to coalesce adjacent hyphens.
    `protected TaggedStrings` | `contractions` List of words starting with & or ' which should not be split.
    `protected java.lang.String` | `contractionsURL` URL for List of words starting with & or ' .
    `protected static java.util.regex.Matcher` | `hyphensMatcher`
    `protected static java.util.regex.Pattern` | `hyphensPattern` Pattern for 2 or more hyphens.
    `protected Logger` | `logger` Logger used for output.
    `protected PreTokenizer` | `preTokenizer` The preTokenizer used here,

  - ### Constructor Summary

    Constructors
    Constructor and Description
    `AbstractWordTokenizer()` Create a word tokenizer.

  - ### Method Summary

    Methods
    Modifier and Type | Method and Description
    --- | ---
    `void` | `addWordToSentence(java.util.List<java.lang.String> sentence, java.lang.String word)` Add word to list of words in sentence.
    `abstract java.util.List<java.lang.String>` | `extractWords(java.lang.String text)` Break text into word tokens.
    `int[]` | `findWordOffsets(java.lang.String sentenceText, java.util.List<?> words)` Find starting offsets of words in a sentence.
    `Logger` | `getLogger()` Get the logger.
    `PreTokenizer` | `getPreTokenizer()` Get the preTokenizer.
    `protected boolean` | `isClosingQuote(char ch)` Is character a closing quote?
    `protected boolean` | `isLetterOrSingleQuote(char ch)` Is character a letter or a single quote?
    `boolean` | `isMultipleHyphens(java.lang.String s)` True if string contains only 2 or more hyphens.
    `boolean` | `isSingleOpeningQuote(char ch)` True if character is a single opening quote.
    `protected void` | `loadContractions()` Load list of non-breakable words and contractions.
    `java.lang.String` | `preprocessToken(java.lang.String token, java.util.List<java.lang.String> tokenList)` Preprocess a word token.
    `void` | `setAbbreviations(Abbreviations abbreviations)` Set abbreviations.
    `void` | `setAposTokens(AposTokens aposTokens)` Set apostophe tokens.
    `void` | `setLogger(Logger logger)` Set the logger.
    `void` | `setPreTokenizer(PreTokenizer preTokenizer)` Set the preTokenizer.
    `protected java.lang.String[]` | `splitToken(java.lang.String token)` Split a token if necessary.

    - ### Methods inherited from class edu.northwestern.at.utils.IsCloseableObject

      `close`

    - ### Methods inherited from class java.lang.Object

      `clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

    - ### Methods inherited from interface edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer.WordTokenizer

      `close`

    - ### Methods inherited from interface edu.northwestern.at.utils.IsCloseable

      `close`

- - ### Field Detail

    - #### preTokenizer

          protected PreTokenizer preTokenizer

      The preTokenizer used here,

    - #### contractions

          protected TaggedStrings contractions

      List of words starting with & or ' which should not be split.

    - #### contractionsURL

          protected java.lang.String contractionsURL

      URL for List of words starting with & or ' .

    - #### logger

          protected Logger logger

      Logger used for output.

    - #### abbreviations

          protected Abbreviations abbreviations

      Abbreviations.

    - #### aposTokens

          protected AposTokens aposTokens

      Apostrophe tokens.

    - #### coalesceHyphens

          protected boolean coalesceHyphens

      True to coalesce adjacent hyphens.

    - #### coalesceAsterisks

          protected boolean coalesceAsterisks

      True to coalesce adjacent asterisks.

    - #### apostropheCanBeQuote

          protected boolean apostropheCanBeQuote

      True if apostrophes can be single quotes.

    - #### hyphensPattern

          protected static final java.util.regex.Pattern hyphensPattern

      Pattern for 2 or more hyphens.

    - #### hyphensMatcher

          protected static final java.util.regex.Matcher hyphensMatcher

  - ### Constructor Detail

    - #### AbstractWordTokenizer

          public AbstractWordTokenizer()

      Create a word tokenizer.

  - ### Method Detail

    - #### getLogger

          public Logger getLogger()

      Get the logger.
      Specified by:
        `getLogger` in interface `UsesLogger`
      Returns:
        The logger.

    - #### setLogger

          public void setLogger(Logger logger)

      Set the logger.
      Specified by:
        `setLogger` in interface `UsesLogger`
      Parameters:
        `logger` - The logger.

    - #### setAbbreviations

          public void setAbbreviations(Abbreviations abbreviations)

      Set abbreviations.
      Specified by:
        `setAbbreviations` in interface `WordTokenizer`
      Parameters:
        `abbreviations` - Abbreviations.

    - #### setAposTokens

          public void setAposTokens(AposTokens aposTokens)

      Set apostophe tokens.
      Specified by:
        `setAposTokens` in interface `WordTokenizer`
      Parameters:
        `aposTokens` - Apostrophe tokens.

    - #### getPreTokenizer

          public PreTokenizer getPreTokenizer()

      Get the preTokenizer.
      Specified by:
        `getPreTokenizer` in interface `WordTokenizer`
      Returns:
        The preTokenizer.

    - #### setPreTokenizer

          public void setPreTokenizer(PreTokenizer preTokenizer)

      Set the preTokenizer.
      Specified by:
        `setPreTokenizer` in interface `WordTokenizer`
      Parameters:
        `preTokenizer` - The preTokenizer.

    - #### loadContractions

          protected void loadContractions()

      Load list of non-breakable words and contractions.

    - #### preprocessToken

          public java.lang.String preprocessToken(java.lang.String token,
                                         java.util.List<java.lang.String> tokenList)

      Preprocess a word token.
      Specified by:
        `preprocessToken` in interface `WordTokenizer`
      Parameters:
        `token` - Token to preprocess.
        `tokenList` - List of previous tokens already issued.
      Returns:
        Preprocessed token. The token list may also have been modified.

    - #### isSingleOpeningQuote

          public boolean isSingleOpeningQuote(char ch)

      True if character is a single opening quote.
      Parameters:
        `ch` - Character to check for being a single opening quote.
      Returns:
        true if character is a single opening quote.

    - #### isLetterOrSingleQuote

          protected boolean isLetterOrSingleQuote(char ch)

      Is character a letter or a single quote?
      Parameters:
        `ch` - Character.
      Returns:
        true if character is a letter or a quote.

    - #### isClosingQuote

          protected boolean isClosingQuote(char ch)

      Is character a closing quote?
      Parameters:
        `ch` - Character.
      Returns:
        true if character is a closing quote.

    - #### splitToken

          protected java.lang.String[] splitToken(java.lang.String token)

      Split a token if necessary.
      Parameters:
        `token` - Token to split.
      Returns:
        The tokens.

    - #### addWordToSentence

          public void addWordToSentence(java.util.List<java.lang.String> sentence,
                               java.lang.String word)

      Add word to list of words in sentence.
      Specified by:
        `addWordToSentence` in interface `WordTokenizer`
      Parameters:
        `sentence` - Result sentence.
        `word` - Word to add.

    - #### findWordOffsets

          public int[] findWordOffsets(java.lang.String sentenceText,
                              java.util.List<?> words)

      Find starting offsets of words in a sentence.
      Specified by:
        `findWordOffsets` in interface `WordTokenizer`
      Parameters:
        `sentenceText` - Text from which tokens were extracted.
        `words` - List of words extracted from sentence text. N.B. If the words aren't from the specified sentence text, the resulting offsets will be meaningless.
      Returns:
        int array of starting offsets in sentenceText for each word. The first offset starts at 0. There is one more offset than the number of words -- the last offset is where the word after the last word would start.

    - #### isMultipleHyphens

          public boolean isMultipleHyphens(java.lang.String s)

      True if string contains only 2 or more hyphens.
      Parameters:
        `s` - String to check for hyphens.
      Returns:
        true if string contains only hyphens and at least two of them.

    - #### extractWords

          public abstract java.util.List<java.lang.String> extractWords(java.lang.String text)

      Break text into word tokens.
      Specified by:
        `extractWords` in interface `WordTokenizer`
      Parameters:
        `text` - Text to break into word tokens.
      Returns:
        List of word tokens.

        Word tokens may be words, numbers, punctuation, etc.