[LANG-220] [lang] Tokenizer Enhancements: reset input string, static CSV/TSV factories

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.0
• Fix Version/s: 2.2

Environment

Operating System: other
Platform: Other

• Bugzilla Id: 26699
• Assignee: Unassigned
• Reporter: Matthew Inger
• Created: 06/Feb/04 04:59
• Updated: 16/May/06 12:29
• Resolved: 16/May/06 12:29

Description

Tokenizer is missing the following features:

1. Reset of the input string to a new value. This would be helpful
when parsing large files, as you could use the same tokenizer instance
on each line of the file by reseting the input on the tokenizer: