Class Ferret::Analysis::TokenStream
In: ext/r_analysis.c
Parent: Object

Summary

A TokenStream enumerates the sequence of tokens, either from fields of a document or from query text.

This is an abstract class. Concrete subclasses are:

Tokenizer:a TokenStream whose input is a string
TokenFilter:a TokenStream whose input is another TokenStream

Methods

next   text   text=  

Public Instance methods

Return the next token from the TokenStream or nil if there are no more tokens.

Return the text that the TokenStream is tokenizing

Set the text attribute of the TokenStream to the text you wish to be tokenized. For example, you may do this;

    token_stream.text = File.read(file_name)

[Validate]