[ARROW-8828] [Rust] Implement SQL tokenizer

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Won't Fix
• Fix Version/s: 2.0.0
• Component/s: Rust, Rust - DataFusion
• External issue URL: https://github.com/apache/arrow/issues/17185
• Assignee: Andy Grove
• Reporter: Andy Grove
• Created: 16/May/20 18:59
• Updated: 11/Jan/23 08:02
• Resolved: 14/Jul/20 13:48

Description

Implement enum for all supported SQL tokens and implement a tokenizer that can tokenize a SQL string and produce a Vec<SQLToken>.