edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer

## Interface WordTokenizer

- All Known Implementing Classes:
    AbstractWordTokenizer, ApostrophesAreNotQuotesWordTokenizer, ContractionTokenizer, DefaultWordTokenizer, EEBOWordTokenizer, ICU4JBreakIteratorWordTokenizer, NoopWordTokenizer, PennTreebankTokenizer, WhitespaceWordTokenizer

      public interface WordTokenizer

  Interface for tokenizing a string into "words".

- - ### Method Summary

    Methods
    Modifier and Type | Method and Description
    --- | ---
    `void` | `addWordToSentence(java.util.List<java.lang.String> sentence, java.lang.String word)` Add word to list of words in sentence.
    `void` | `close()` Close down the word tokenizer.
    `java.util.List<java.lang.String>` | `extractWords(java.lang.String text)` Break text into word tokens.
    `int[]` | `findWordOffsets(java.lang.String sentenceText, java.util.List<?> words)` Find starting offsets of words in a sentence.
    `PreTokenizer` | `getPreTokenizer()` Get the preTokenizer.
    `java.lang.String` | `preprocessToken(java.lang.String token, java.util.List<java.lang.String> tokenList)` Preprocess a word token.
    `void` | `setAbbreviations(Abbreviations abbreviations)` Set abbreviations.
    `void` | `setAposTokens(AposTokens aposTokens)` Set apostophe tokens.
    `void` | `setPreTokenizer(PreTokenizer preTokenizer)` Set the preTokenizer.

- - ### Method Detail

    - #### getPreTokenizer

          PreTokenizer getPreTokenizer()

      Get the preTokenizer.
      Returns:
        The preTokenizer.

    - #### setPreTokenizer

          void setPreTokenizer(PreTokenizer preTokenizer)

      Set the preTokenizer.
      Parameters:
        `preTokenizer` - The preTokenizer.

    - #### setAbbreviations

          void setAbbreviations(Abbreviations abbreviations)

      Set abbreviations.
      Parameters:
        `abbreviations` - Abbreviations.

    - #### setAposTokens

          void setAposTokens(AposTokens aposTokens)

      Set apostophe tokens.
      Parameters:
        `aposTokens` - Apostrophe tokens.

    - #### addWordToSentence

          void addWordToSentence(java.util.List<java.lang.String> sentence,
                               java.lang.String word)

      Add word to list of words in sentence.
      Parameters:
        `sentence` - Result sentence.
        `word` - Word to add.

    - #### extractWords

          java.util.List<java.lang.String> extractWords(java.lang.String text)

      Break text into word tokens.
      Parameters:
        `text` - Text to break into word tokens.
      Returns:
        List of word tokens.

        Word tokens may be words, numbers, punctuation, etc.

    - #### findWordOffsets

          int[] findWordOffsets(java.lang.String sentenceText,
                              java.util.List<?> words)

      Find starting offsets of words in a sentence.
      Parameters:
        `sentenceText` - Text from which tokens were extracted.
        `words` - List of words extracted from sentence text. N.B. If the words aren't from the specified sentence text, the resulting offsets will be meaningless.
      Returns:
        int array of starting offsets in sentenceText for each word. The first offset starts at 0. There is one more offset than the number of words -- the last offset is where the word after the last word would start.

    - #### preprocessToken

          java.lang.String preprocessToken(java.lang.String token,
                                         java.util.List<java.lang.String> tokenList)

      Preprocess a word token.
      Parameters:
        `token` - Token to preprocess.
        `tokenList` - List of previous tokens already issued.
      Returns:
        Preprocessed token. The token list may also have been modified.

    - #### close

          void close()

      Close down the word tokenizer.