edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer

## Class DefaultWordTokenizer

- java.lang.Object
- - edu.northwestern.at.utils.IsCloseableObject
  - - edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer.AbstractWordTokenizer
    - - edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer.DefaultWordTokenizer

- All Implemented Interfaces:
    WordTokenizer, IsCloseable, UsesLogger
  Direct Known Subclasses:
    ApostrophesAreNotQuotesWordTokenizer, EEBOWordTokenizer

      public class DefaultWordTokenizer
      extends AbstractWordTokenizer
      implements WordTokenizer

  Default word tokenizer.

- - ### Field Summary

    - ### Fields inherited from class edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer.AbstractWordTokenizer

      `abbreviations, aposTokens, apostropheCanBeQuote, coalesceAsterisks, coalesceHyphens, contractions, contractionsURL, hyphensMatcher, hyphensPattern, logger, preTokenizer`

  - ### Constructor Summary

    Constructors
    Constructor and Description
    `DefaultWordTokenizer()` Create a simple word tokenizer.

  - ### Method Summary

    Methods
    Modifier and Type | Method and Description
    --- | ---
    `void` | `addWordToSentence(java.util.List<java.lang.String> sentence, java.lang.String word)` Add word to list of words in sentence.
    `java.util.List<java.lang.String>` | `extractWords(java.lang.String text)` Break text into word tokens.

    - ### Methods inherited from class edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer.AbstractWordTokenizer

      `findWordOffsets, getLogger, getPreTokenizer, isClosingQuote, isLetterOrSingleQuote, isMultipleHyphens, isSingleOpeningQuote, loadContractions, preprocessToken, setAbbreviations, setAposTokens, setLogger, setPreTokenizer, splitToken`

    - ### Methods inherited from class edu.northwestern.at.utils.IsCloseableObject

      `close`

    - ### Methods inherited from class java.lang.Object

      `clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

    - ### Methods inherited from interface edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer.WordTokenizer

      `close, findWordOffsets, getPreTokenizer, preprocessToken, setAbbreviations, setAposTokens, setPreTokenizer`

    - ### Methods inherited from interface edu.northwestern.at.utils.IsCloseable

      `close`

- - ### Constructor Detail

    - #### DefaultWordTokenizer

          public DefaultWordTokenizer()

      Create a simple word tokenizer.

  - ### Method Detail

    - #### extractWords

          public java.util.List<java.lang.String> extractWords(java.lang.String text)

      Break text into word tokens.
      Specified by:
        `extractWords` in interface `WordTokenizer`
      Specified by:
        `extractWords` in class `AbstractWordTokenizer`
      Parameters:
        `text` - Text to break into word tokens.
      Returns:
        List of word tokens.

        Word tokens may be words, numbers, punctuation, etc.

    - #### addWordToSentence

          public void addWordToSentence(java.util.List<java.lang.String> sentence,
                               java.lang.String word)

      Add word to list of words in sentence.
      Specified by:
        `addWordToSentence` in interface `WordTokenizer`
      Overrides:
        `addWordToSentence` in class `AbstractWordTokenizer`
      Parameters:
        `sentence` - Result sentence.
        `word` - Word to add.