edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer

Class TokenizerUtils

java.lang.Object

edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer.TokenizerUtils

public class TokenizerUtils
extends java.lang.Object

Tokenizer utilities.

Field Summary

Fields

Modifier and Type	Field and Description
protected static Abbreviations	abbreviations

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	TokenizerUtils() Don't allow instantiation, do allow overrides.

Method Summary

Methods

Modifier and Type	Method and Description
static java.lang.String	getTokenType(java.lang.String token) Get token type.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

abbreviations

protected static Abbreviations abbreviations

Constructor Detail

TokenizerUtils

protected TokenizerUtils()

Don't allow instantiation, do allow overrides.

Method Detail

getTokenType

public static java.lang.String getTokenType(java.lang.String token)

Get token type.

Parameters:

token - The token.

Returns:

Token type (plain, abbreviation, punctuation, etc.)