[FLINK-26360] [Umbrella] Improvement for Hive Query Syntax Compatibility

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.17.0
• Component/s: Connectors / Hive
• Assignee: luoyuxia
• Reporter: luoyuxia
• Created: 25/Feb/22 02:00
• Updated: 17/Jan/23 05:42
• Resolved: 19/Dec/22 07:30

Description

Currently, we have a support for hive synatax compatibility in flink as described in FLINK-21529.

But there're still some features we don't support or some other issues when using hive synatax.

In here, we want to make a improvement to solve the issues encountered when using Hive dialect to make it be more smoothly when you mrigate your hive job to flink or enable you write flink job using hive synatax with less knowledge about flink sql.

Feel free to leave your comment.

Issue Links

• relates to: FLINK-26603 [Umbrella] Decouple Hive with Flink planner

Sub-Tasks

• "Unexpected correlate variable $cor0" when using hive dialect to write a subquery: Closed
• "IndexOutOfBoundsException" when subquery select all field from using hive dialect: Closed
• Fail to use such expression like if(1= 1, 2, 3): Resolved
• Improve hive udtf: Resolved
• improve hive udaf: Resolved
• Support "insert directory": Closed
• Support read hive variables using the syntax like select ${hiveconf:select_time}: Closed
• support hive transform using syntax: Closed
• Hive dialect supports CLUSTERED BY/ SORTED BY table: Resolved
• Hive dialect supports "CREATE FUNCTION USING xxx.jar": Closed
• Hive dialect support "LOAD DATA INPATH": Closed
• Hive dialect supports "MACRO": Closed
• Fail to call hive udf when secondary parameter required constant but pass value like "-1": Closed
• Fix IndexOutOfBoundsException for non equality condition exists in left semi join in Hive dialect: Closed
• Support handle join involving complex types in on condition: Closed
• Fix "schema of both sides of union should match" exception when one side of union select multiple same coulmns: Closed
• Fix incomparable exception between boolean type and numeric type in Hive dialect: Closed
• Hive dialect supports IS_DISTINCT_FROM: Closed
• Fix exception for casting timestamp to decimal in Hive dialect: Closed
• Improve supporting for complex data type for Hive dialect: Closed
• Fix divide by zero exception in Hive dialect: Closed
• Supports read Hive table partitioned by decimal type: Closed
• Fail to call Hive UDAF when the UDAF is with only one parameter with array type: Closed
• Support partition filter push down when there exists default_parition: Closed
• Support subdirectories with Hive tables: Closed
• Hive dialect supports "select current_database()": Closed
• Fix fail to count mutiple fields excpetion in Hive dialect: Closed
• Hive dialect support INTERVAL type: Closed
• HiveInspectors should use correct writable type to creaet ConstantObjectInspector: Closed
• Hive dialect support "tablesample (xx rows)": Closed
• Hive dialect support set variable: Closed
• IndexOutOfBoundsException is thrown when project contains window which dosen't refer all fields of input when using Hive dialect: Closed
• Exception will be thrown when over window contains grouping in Hive Dialect: Resolved
• Hive dialect supports add jar: Closed
• Hive dialect: `insert overwrite table partition if not exists` will throw exception when tablename is like 'database.table': Closed
• Introduce operation execution plugin: Closed
• HiveFunctionWrapper should use UserClassLoader to load class: Resolved
• Support Hive's UDAF which implement GenericUDAFResolver: Closed
• Flink hive parser considers literal floating point number differently than Hive SQL: Resolved