Uploaded image for project: 'Apache NiFi'
  1. Apache NiFi
  2. NIFI-4473

Add support for large result sets and normalizing Avro names to SelectHiveQL

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.5.0
    • Component/s: Extensions
    • Labels:
      None

      Description

      A number of enhancements were made to processors like QueryDatabaseTable to allow for such things as:

      • Splitting result sets into multiple flow files (i.e. Max Rows Per Flowfile property)
      • Max number of splits/rows returned (Max fragments)
      • Normalizing names to be Avro-compatible

      The RDBMS processors also now support Avro logical types, but the version of Avro needed by the current version of Hive (1.2.1) is Avro 1.7.7, which does not support logical types.

      These enhancements were made to JdbcCommon, but not to HiveJdbcCommon (the Hive version of the JDBC utils class). Since Hive queries can return even larger result sets than traditional RDBMS, these properties/enhancements are at least as valuable to have for SelectHiveQL.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                mattyb149 Matt Burgess
                Reporter:
                mattyb149 Matt Burgess
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: