edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer

Class EccoPreTokenizer

java.lang.Object

edu.northwestern.at.utils.IsCloseableObject

edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer.AbstractPreTokenizer

edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer.EccoPreTokenizer

All Implemented Interfaces:

PreTokenizer, UsesLogger

public class EccoPreTokenizer
extends AbstractPreTokenizer
implements PreTokenizer

A pretokenizer for ECCO texts.

Field Summary

Fields

Modifier and Type	Field and Description
protected static PatternReplacer	doubleBackTicksReplacer Double back-ticks.
protected static java.lang.String	EccoAlwaysSeparators
protected static PatternReplacer	singleBackTicksReplacer Single back-tick followed by a capital letter.
protected static PatternReplacer	wordOrSpanGapReplacer Word or span gap.

Fields inherited from class edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer.AbstractPreTokenizer

alwaysSeparators, alwaysSeparatorsReplacer, asterisks, commaSeparator, commaSeparatorReplacer, hyphens, logger, periods

Constructor Summary

Constructors

Constructor and Description
EccoPreTokenizer() Create an Ecco pretokenizer.

Method Summary

Methods

Modifier and Type	Method and Description
java.lang.String	pretokenize(java.lang.String line) Prepare text for tokenization.

Methods inherited from class edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer.AbstractPreTokenizer

getLogger, setLogger

Methods inherited from class edu.northwestern.at.utils.IsCloseableObject

close

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer.PreTokenizer

close

Field Detail

EccoAlwaysSeparators

protected static final java.lang.String EccoAlwaysSeparators

See Also:

Constant Field Values

wordOrSpanGapReplacer

protected static final PatternReplacer wordOrSpanGapReplacer

Word or span gap.

doubleBackTicksReplacer

protected static final PatternReplacer doubleBackTicksReplacer

Double back-ticks.

singleBackTicksReplacer

protected static final PatternReplacer singleBackTicksReplacer

Single back-tick followed by a capital letter.

Constructor Detail

EccoPreTokenizer

public EccoPreTokenizer()

Create an Ecco pretokenizer.

Method Detail

pretokenize

public java.lang.String pretokenize(java.lang.String line)

Prepare text for tokenization.

Specified by:

pretokenize in interface PreTokenizer

Overrides:

pretokenize in class AbstractPreTokenizer

Parameters:

line - The text to prepare for tokenization,

Returns:

The pretokenized text.