edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer

Class EccoPostTokenizer

java.lang.Object

edu.northwestern.at.utils.IsCloseableObject

edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer.AbstractPostTokenizer

edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer.DefaultPostTokenizer

edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer.EccoPostTokenizer

All Implemented Interfaces:

PostTokenizer, UsesLogger

public class EccoPostTokenizer
extends DefaultPostTokenizer
implements PostTokenizer

Ecco text post tokenizer which processes tokens after extraction.

Field Summary

Fields

Modifier and Type	Field and Description
protected static java.util.regex.Matcher	eccoPostMatcher
protected static java.util.regex.Pattern	eccoPostPattern Character replacer.

Fields inherited from class edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer.AbstractPostTokenizer

logger

Constructor Summary

Constructors

Constructor and Description
EccoPostTokenizer() Create Ecco text postTokenizer.

Method Summary

Methods

Modifier and Type	Method and Description
java.lang.String[]	postTokenize(java.lang.String token) Process a token after tokenization.

Methods inherited from class edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer.AbstractPostTokenizer

getLogger, setLogger

Methods inherited from class edu.northwestern.at.utils.IsCloseableObject

close

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer.PostTokenizer

close

Field Detail

eccoPostPattern

protected static java.util.regex.Pattern eccoPostPattern

Character replacer.

eccoPostMatcher

protected static java.util.regex.Matcher eccoPostMatcher

Constructor Detail

EccoPostTokenizer

public EccoPostTokenizer()

Create Ecco text postTokenizer.

Method Detail

postTokenize

public java.lang.String[] postTokenize(java.lang.String token)

Process a token after tokenization.

Specified by:

postTokenize in interface PostTokenizer

Overrides:

postTokenize in class DefaultPostTokenizer

Parameters:

token - The token to process after tokenization.

Returns:

Array of two strings. [0] = the token minimally processed. [1] = the token maximally processed.

The minimally processed token is typically results in an original spelling.

The maximally processed token typically results in a partially or completely standardized spelling.

These may be identical.