[PIG-1618] Switch to new parser generator technology

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.8.0
• Fix Version/s: 0.9.0
• Assignee: Xuefu Zhang
• Reporter: Alan Gates
• Created: 17/Sep/10 15:55
• Updated: 02/May/13 02:29
• Resolved: 18/Mar/11 00:34

Description

There are many bugs in Pig related to the parser, particularly to bad error messages. After review of Java CC we feel these will be difficult to address using that tool. Also, the .jjt files used by JavaCC are hard to understand and maintain.

ANTLR is being reviewed as the most likely choice to move to, but other parsers will be reviewed as well.

This JIRA will act as an umbrella issue for other parser issues.

Issue Links

• is blocked by: PIG-1622 DEFINE streaming options are ill defined and not properly documented
• is depended upon by: PIG-217 Syntax Errors
• is depended upon by: PIG-394 Syntax for ?: requires parens in FOREACH
• is depended upon by: PIG-579 Adding newlines to format foreach statement with constants causes parse errors
• is depended upon by: PIG-838 Parser does not handle ctrl-m ('\u000d') as argument to PigStorage
• is depended upon by: PIG-900 ORDER BY syntax wrt parentheses is somewhat different than GROUP BY and FILTER BY
• is depended upon by: PIG-1339 International characters in column names not supported
• is depended upon by: PIG-19 A=load causes parse error
• is depended upon by: PIG-356 map lookup on empty key should be disallowed at parse time
• is depended upon by: PIG-377 Grunt parser doesn't handle escape codes correctly
• is depended upon by: PIG-548 ParseException involving "as" keyword
• is depended upon by: PIG-583 Bag constants used in non foreach statements cause lexical errors
• is depended upon by: PIG-1623 Register syntax is ambiguous
• is depended upon by: PIG-1663 Order by only allows ordering on columns, not expressions
• is depended upon by: PIG-731 Passing semicolon as a parameter in UDF causes parser error
• is depended upon by: PIG-596 Anonymous tuples in bags create ParseExceptions
• is depended upon by: PIG-618 Bad error message when period rather than comma appears as separator in UDF parameter list
• is depended upon by: PIG-1152 bincond operator throws parser error
• is depended upon by: PIG-1462 No informative error message on parse problem
• is depended upon by: PIG-1545 Secondary alias gives problem, when it has alias in the group by statement.
• is depended upon by: PIG-1774 parser should not require bincond/conditional operator to be within parenthesis
• is depended upon by: PIG-144 The error message should be more meaningful when there is a typo in PIg script
• is depended upon by: PIG-670 DIFF contains an invalid expression - possible parser error
• is depended upon by: PIG-674 Improve errors in Pig parser
• is depended upon by: PIG-719 store <expr> into 'filename'; should be valid syntax, but does not work
• is depended upon by: PIG-1092 Pig Latin Parser fails to recognize "\n" as a whitespace
• is depended upon by: PIG-1581 Parser fails to recognize semicolons in quoted strings
• is depended upon by: PIG-1619 Bad error message when a double constant is incorrectly specified
• is depended upon by: PIG-1620 ARRANGE keyword should be deprecated
• is depended upon by: PIG-1621 What does EVAL keyword do?
• is depended upon by: PIG-1244 parameter syntax in scripts, add support for ${VAR} (in addition to current $VAR)
• is depended upon by: PIG-1499 Type error message does not include complex type
• is depended upon by: PIG-1387 Syntactical Sugar for PIG-1385

Sub-Tasks

• as_clause in foreach statement should differentiate between simple type and type within tuple: Resolved
• new parser : duplicate ConstantExpressions created in inner plan: Closed