Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
None
-
None
Description
NIFI-5475 upgraded the version of Hive 3 to Apache Hive 3.1.0, and some code changes had to be made as there were new Writable types to be used in place of the old ones. As a result of the upgrade, NIFI-5491 was discovered and included some fixes for PutHive3Streaming to support various primitive data types as well as structs.
However it appears that NIFI-5491 did not cover all the available Hive types, and NIFI-5475's changes were to PutORC for time/date types, but similar changes have been made to the Hive writer as well.
PutHive3Streaming should support all available Hive types where prudent (i.e. where NiFi Record Field Types can be converted to Hive Column Types). The current list of "top-level" types include:
PRIMITIVE, LIST, MAP, STRUCT, UNION
And the current list of PRIMITIVE types include:
VOID, BOOLEAN, BYTE, SHORT, INT, LONG, FLOAT, DOUBLE, STRING,
DATE, TIMESTAMP, TIMESTAMPLOCALTZ, BINARY, DECIMAL, VARCHAR, CHAR,
INTERVAL_YEAR_MONTH, INTERVAL_DAY_TIME, UNKNOWN
As of NIFI-5475 I believe PutHive3Streaming supports BOOLEAN, BYTE, SHORT, INT, LONG, FLOAT, DOUBLE, STRING, VARCHAR, and CHAR for primitive types, and with respect to the supported primitive types, PutHive3Streaming supports LIST, MAP, STRUCT, and UNION.
The remaining list of primitive types is DATE, TIMESTAMP, BINARY, and DECIMAL. VOID and UNKNOWN are used "under the hood" I believe, and ORC doesn't currently support TIMESTAMPLOCALTZ. INTERVAL_YEAR_MONTH and INTERVAL_DAY_TIME are only valid for queries and not as column types.
