[NIFI-4473] Add support for large result sets and normalizing Avro names to SelectHiveQL

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.5.0
• Component/s: Extensions
• Assignee: Matt Burgess
• Reporter: Matt Burgess
• Created: 09/Oct/17 12:49
• Updated: 04/Apr/18 19:22
• Resolved: 03/Nov/17 16:05

Description

A number of enhancements were made to processors like QueryDatabaseTable to allow for such things as:

• Splitting result sets into multiple flow files (i.e. Max Rows Per Flowfile property)
• Max number of splits/rows returned (Max fragments)
• Normalizing names to be Avro-compatible

The RDBMS processors also now support Avro logical types, but the version of Avro needed by the current version of Hive (1.2.1) is Avro 1.7.7, which does not support logical types.

These enhancements were made to JdbcCommon, but not to HiveJdbcCommon (the Hive version of the JDBC utils class). Since Hive queries can return even larger result sets than traditional RDBMS, these properties/enhancements are at least as valuable to have for SelectHiveQL.

Issue Links

• causes: NIFI-4862 Copy original FlowFile attributes to output FlowFiles at SelectHiveQL processor
• links to: GitHub Pull Request #2212
• links to: GitHub Pull Request #2215
• links to: GitHub Pull Request #2247