edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer

Class DefaultPostTokenizer

java.lang.Object

edu.northwestern.at.utils.IsCloseableObject

edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer.AbstractPostTokenizer

edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer.DefaultPostTokenizer

All Implemented Interfaces:

PostTokenizer, UsesLogger

Direct Known Subclasses:

EccoPostTokenizer

public class DefaultPostTokenizer
extends AbstractPostTokenizer
implements PostTokenizer

Default post tokenizer which processes tokens after extraction.

Field Summary

Fields inherited from class edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer.AbstractPostTokenizer

logger

Constructor Summary

Constructors

Constructor and Description
DefaultPostTokenizer() Create a default postTokenizer.

Method Summary

Methods

Modifier and Type	Method and Description
java.lang.String[]	postTokenize(java.lang.String token) Process a token after tokenization.

Methods inherited from class edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer.AbstractPostTokenizer

getLogger, setLogger

Methods inherited from class edu.northwestern.at.utils.IsCloseableObject

close

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer.PostTokenizer

close

Constructor Detail

DefaultPostTokenizer

public DefaultPostTokenizer()

Create a default postTokenizer.

Method Detail

postTokenize

public java.lang.String[] postTokenize(java.lang.String token)

Process a token after tokenization.

Specified by:

postTokenize in interface PostTokenizer

Overrides:

postTokenize in class AbstractPostTokenizer

Parameters:

token - The token to process after tokenization.

Returns:

Array of two strings. [0] = the token minimally processed. [1] = the token maximally processed.

The minimally processed token is typically results in an original spelling.

The maximally processed token typically results in a partially or completely standardized spelling.

These may be identical.