edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer

Class NoopPreTokenizer

java.lang.Object

edu.northwestern.at.utils.IsCloseableObject

edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer.AbstractPreTokenizer

edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer.NoopPreTokenizer

All Implemented Interfaces:

PreTokenizer, UsesLogger

public class NoopPreTokenizer
extends AbstractPreTokenizer
implements PreTokenizer

A "no-op" preTokenizer which leaves the input string unchanged.

Field Summary

Fields inherited from class edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer.AbstractPreTokenizer

alwaysSeparators, alwaysSeparatorsReplacer, asterisks, commaSeparator, commaSeparatorReplacer, hyphens, logger, periods

Constructor Summary

Constructors

Constructor and Description
NoopPreTokenizer() Create a "no-op" preTokenizer.

Method Summary

Methods

Modifier and Type	Method and Description
java.lang.String	pretokenize(java.lang.String line) Prepares a string for tokenization.

Methods inherited from class edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer.AbstractPreTokenizer

getLogger, setLogger

Methods inherited from class edu.northwestern.at.utils.IsCloseableObject

close

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer.PreTokenizer

close

Constructor Detail

NoopPreTokenizer

public NoopPreTokenizer()

Create a "no-op" preTokenizer.

Method Detail

pretokenize

public java.lang.String pretokenize(java.lang.String line)

Prepares a string for tokenization.

Specified by:

pretokenize in interface PreTokenizer

Overrides:

pretokenize in class AbstractPreTokenizer

Parameters:

line - The string to prepare for tokenization.

Returns:

The input string with no changes.