[CALCITE-4474] SqlSimpleParser inner Tokenizer should not recognize the sql of TokenType.ID or some keywords in some case

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.26.0
• Fix Version/s: 1.27.0
• Component/s: core
• Labels: pull-request-available
• Assignee: Unassigned
• Reporter: wangjie
• Created: 21/Jan/21 16:06
• Updated: 03/Jun/21 22:32
• Resolved: 28/May/21 15:10

Description

in SqlSimpleParser.Tokenizer#nextToken() lines 401 - 423, is used to recognize the sql of TokenType.ID or some keywords. 

If a certain segment of characters is continuously composed of Token,the function of this code may be wrong.

 

E.g :
(1)select * from a where price> 10.0-–comment

【10.0-–comment】should be recognize as TokenType.ID("10.0") and TokenType.COMMENT, but it recognize as TokenType.ID("10.0--comment")
(2)select * from a where column_b='/* this is not comment */'

【/* this is not comment */】should be recognize as TokenType.SQID("/* this is not comment */"), but it was not

  

you can see reproduce-demo in https://github.com/wangjie-fourth/calcite-reproduce

Issue Links

• links to: GitHub Pull Request #2332