edu.stanford.nlp.simple

Class Token

java.lang.Object

edu.stanford.nlp.simple.Token

All Implemented Interfaces:

AbstractToken, HasContext, HasIndex, HasLemma, HasNER, HasOffset, HasOriginalText, HasTag, HasWord, java.io.Serializable

public class Token
extends java.lang.Object
implements AbstractToken

A utility class for representing a token in the Simple API. This nominally tries to conform to a CoreLabel-like interface, though many of the methods are not supported (most notably, the setters).

Author:

Gabor Angeli

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
int	index The index of this token in the underlying sentence.
Sentence	sentence The underlying sentence supplying the fields for this token.

Constructor Summary

Constructors

Constructor and Description
Token(Sentence sentence, int index) Create a wrapper for a token, given a sentence and an index in the sentence

Method Summary

Modifier and Type	Method and Description
java.lang.String	after() Return the whitespace String after the word.
java.lang.String	before()
int	beginPosition() Return the beginning char offset of the label (or -1 if none).
int	characterOffsetBegin()
int	characterOffsetEnd()
java.lang.String	docID()
int	endPosition() Return the ending char offset of the label (or -1 if none).
java.util.Optional<java.lang.Integer>	governor()
java.util.Optional<java.lang.String>	incomingDependencyLabel()
int	index()
java.lang.String	lemma() Return the lemma value of the label (or null if none).
java.lang.String	ner() Return the named entity class of the label (or null if none).
java.lang.String	nerTag()
Token	next() The next token in the sentence.
java.lang.String	originalText() Return the String which is the original character sequence of the token.
java.lang.String	posTag()
Token	previous() The previous token in the sentence.
int	sentIndex()
void	setAfter(java.lang.String after) Set the whitespace String after the word.
void	setBefore(java.lang.String before) Set the whitespace String before the word.
void	setBeginPosition(int beginPos) Set the beginning character offset for the label.
void	setDocID(java.lang.String docID)
void	setEndPosition(int endPos) Set the ending character offset of the label (or -1 if none).
void	setIndex(int index)
void	setLemma(java.lang.String lemma) Set the lemma value for the label (if one is stored).
void	setNER(java.lang.String ner) Set the named entity class of the label.
void	setOriginalText(java.lang.String originalText) Set the String which is the original character sequence of the token.
void	setSentIndex(int sentIndex)
void	setTag(java.lang.String tag) Set the tag value for the label (if one is stored).
void	setWord(java.lang.String word) Set the word value for the label (if one is stored).
java.lang.String	tag() Return the tag value of the label (or null if none).
java.lang.String	word() Return the word value of the label (or null if none).

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

sentence

public final Sentence sentence

The underlying sentence supplying the fields for this token.

index

public final int index

The index of this token in the underlying sentence. This can be out of bounds; the sentence is assumed to be infinitely padded.

Constructor Detail

Token

public Token(Sentence sentence,
             int index)

Create a wrapper for a token, given a sentence and an index in the sentence

Method Detail

previous

public Token previous()

The previous token in the sentence.

next

public Token next()

The next token in the sentence.

word

public java.lang.String word()

Description copied from interface: HasWord

Return the word value of the label (or null if none).

Specified by:

word in interface HasWord

Returns:

String the word value for the label

See Also:

Sentence.word(int)

setWord

public void setWord(java.lang.String word)

Description copied from interface: HasWord

Set the word value for the label (if one is stored).

Specified by:

setWord in interface HasWord

Parameters:

word - The word value for the label

originalText

public java.lang.String originalText()

Description copied from interface: HasContext

Return the String which is the original character sequence of the token.

Specified by:

originalText in interface HasContext

Specified by:

originalText in interface HasOriginalText

Returns:

The original character sequence of the token

See Also:

Sentence.originalText(int)

setOriginalText

public void setOriginalText(java.lang.String originalText)

Description copied from interface: HasContext

Set the String which is the original character sequence of the token.

Specified by:

setOriginalText in interface HasContext

Specified by:

setOriginalText in interface HasOriginalText

Parameters:

originalText - The original character sequence of the token

lemma

public java.lang.String lemma()

Description copied from interface: HasLemma

Return the lemma value of the label (or null if none).

Specified by:

lemma in interface HasLemma

Returns:

String the lemma value for the label

See Also:

Sentence.lemma(int)

setLemma

public void setLemma(java.lang.String lemma)

Description copied from interface: HasLemma

Set the lemma value for the label (if one is stored).

Specified by:

setLemma in interface HasLemma

Parameters:

lemma - The lemma value for the label

ner

public java.lang.String ner()

Description copied from interface: HasNER

Return the named entity class of the label (or null if none).

Specified by:

ner in interface HasNER

Returns:

The NER class for the label

See Also:

Sentence.nerTag(int)

setNER

public void setNER(java.lang.String ner)

Description copied from interface: HasNER

Set the named entity class of the label.

Specified by:

setNER in interface HasNER

Parameters:

ner - The NER class for the label

nerTag

public java.lang.String nerTag()

See Also:

Sentence.nerTag(int)

tag

public java.lang.String tag()

Description copied from interface: HasTag

Return the tag value of the label (or null if none).

Specified by:

tag in interface HasTag

Returns:

String the tag value for the label

See Also:

Sentence.posTag(int)

setTag

public void setTag(java.lang.String tag)

Description copied from interface: HasTag

Set the tag value for the label (if one is stored).

Specified by:

setTag in interface HasTag

Parameters:

tag - The tag value for the label

posTag

public java.lang.String posTag()

See Also:

Sentence.posTag(int)

governor

public java.util.Optional<java.lang.Integer> governor()

See Also:

Sentence.governor(int)

characterOffsetBegin

public int characterOffsetBegin()

See Also:

Sentence.characterOffsetBegin(int)

characterOffsetEnd

public int characterOffsetEnd()

See Also:

Sentence.characterOffsetEnd(int)

before

public java.lang.String before()

Specified by:

before in interface HasContext

Returns:

the String before the word

See Also:

Sentence.before(int)

setBefore

public void setBefore(java.lang.String before)

Description copied from interface: HasContext

Set the whitespace String before the word.

Specified by:

setBefore in interface HasContext

Parameters:

before - the whitespace String before the word

after

public java.lang.String after()

Description copied from interface: HasContext

Return the whitespace String after the word.

Specified by:

after in interface HasContext

Returns:

The whitespace String after the word

See Also:

Sentence.after(int)

setAfter

public void setAfter(java.lang.String after)

Description copied from interface: HasContext

Set the whitespace String after the word.

Specified by:

setAfter in interface HasContext

Parameters:

after - The whitespace String after the word

incomingDependencyLabel

public java.util.Optional<java.lang.String> incomingDependencyLabel()

See Also:

Sentence.incomingDependencyLabel(int)

docID

public java.lang.String docID()

Specified by:

docID in interface HasIndex

setDocID

public void setDocID(java.lang.String docID)

Specified by:

setDocID in interface HasIndex

sentIndex

public int sentIndex()

Specified by:

sentIndex in interface HasIndex

setSentIndex

public void setSentIndex(int sentIndex)

Specified by:

setSentIndex in interface HasIndex

index

public int index()

Specified by:

index in interface HasIndex

setIndex

public void setIndex(int index)

Specified by:

setIndex in interface HasIndex

beginPosition

public int beginPosition()

Description copied from interface: HasOffset

Return the beginning char offset of the label (or -1 if none). Note that these are currently measured in terms of UTF-16 char offsets, not codepoints, so that when non-BMP Unicode characters are present, such a character will add 2 to the position. On the other hand, these values will work with String#substring() and you can then calculate the number of codepoints in a substring.

Specified by:

beginPosition in interface HasOffset

Returns:

the beginning position for the label

setBeginPosition

public void setBeginPosition(int beginPos)

Description copied from interface: HasOffset

Set the beginning character offset for the label. Setting this key to "-1" can be used to indicate no valid value.

Specified by:

setBeginPosition in interface HasOffset

Parameters:

beginPos - The beginning position

endPosition

public int endPosition()

Description copied from interface: HasOffset

Return the ending char offset of the label (or -1 if none). As usual in Java, this is the offset of the char after this token. Note that these are currently measured in terms of UTF-16 char offsets, not codepoints, so that when non-BMP Unicode characters are present, such a character will add 2 to the position. On the other hand, these values will work with String#substring() and you can then calculate the number of codepoints in a substring.

Specified by:

endPosition in interface HasOffset

Returns:

the end position for the label

setEndPosition

public void setEndPosition(int endPos)

Description copied from interface: HasOffset

Set the ending character offset of the label (or -1 if none).

Specified by:

setEndPosition in interface HasOffset

Parameters:

endPos - The end character offset for the label