[HIVE-26751] Bug Fixes and Improvements for 3.2.0 release

• Type: Improvement
• Status: Open
• Priority: Blocker
• Resolution: Unresolved
• Affects Version/s: 3.1.3
• Labels: hive-3.2.0-must pull-request-available release-3.2.0
• Target Version/s: 3.2.0
• Assignee: Aman Raj
• Reporter: Aman Raj
• Created: 17/Nov/22 07:00
• Updated: 16/Sep/23 07:59

Description

Creating subtask to track the bug fixes that will go as part of 3.2.0 release.

Issue Links

• contains: HIVE-21291 Restore historical way of handling timestamps in Avro while keeping the new semantics at the same time
• contains: HIVE-22498 Schema tool enhancements to merge catalogs
• contains: HIVE-17917 VectorizedOrcAcidRowBatchReader.computeOffsetAndBucket optimization
• contains: HIVE-25446 Wrong exception thrown if capacity<=0
• contains: HIVE-25247 Fix TestWMMetricsWithTrigger
• contains: HIVE-27678 Backport HIVE-26127 to branch-3
• contains: HIVE-20437 Handle schema evolution from float, double and decimal
• contains: HIVE-22113 Prevent LLAP shutdown on AMReporter related RuntimeException
• contains: HIVE-23889 Empty bucket files are inserted with invalid schema after HIVE-21784
• contains: HIVE-26082 Upgrade DataNucleus dependency to 5.2.8
• contains: HIVE-22208 Column name with reserved keyword is unescaped when query including join on table with mask column is re-written
• contains: HIVE-18284 NPE when inserting data with 'distribute by' clause with dynpart sort optimization
• contains: HIVE-19416 Create single version transactional table metastore statistics for aggregation queries
• contains: HIVE-19661 switch Hive UDFs to use Re2J regex engine
• contains: HIVE-20778 Join reordering may not be triggered if all joins in plan are created by decorrelation logic
• contains: HIVE-21167 Bucketing: Bucketing version 1 is incorrectly partitioning data
• contains: HIVE-21729 Arrow serializer sometimes shifts timestamp by one second
• contains: HIVE-21793 CBO retrieves column stats even if hive.stats.fetch.column.stats is set to false
• contains: HIVE-21862 ORC ppd produces wrong result with timestamp
• contains: HIVE-21866 LLAP status service driver may get stuck with wrong Yarn app ID
• contains: HIVE-21915 Hive with TEZ UNION ALL and UDTF results in data loss
• contains: HIVE-22163 CBO: Enabling CBO turns on stats estimation, even when the estimation is disabled
• contains: HIVE-22170 from_unixtime and unix_timestamp should use user session time zone
• contains: HIVE-22219 Bringing a node manager down blocks restart of LLAP service
• contains: HIVE-22275 OperationManager.queryIdOperation does not properly clean up multiple queryIds
• contains: HIVE-22331 unix_timestamp without argument returns timestamp in millisecond instead of second.
• contains: HIVE-22360 MultiDelimitSerDe returns wrong results in last column when the loaded file has more columns than those in table schema
• contains: HIVE-22405 Add ColumnVector support for ProlepticCalendar
• contains: HIVE-22429 Migrated clustered tables using bucketing_version 1 on hive 3 uses bucketing_version 2 for inserts
• contains: HIVE-22582 Avoid reading table as ACID when table name is starting with "delta" , but table is not transactional and BI Split Strategy is used
• contains: HIVE-22589 Add storage support for ProlepticCalendar in ORC, Parquet, and Avro
• contains: HIVE-22840 Race condition in formatters of TimestampColumnVector and DateColumnVector
• contains: HIVE-24039 Update jquery version to mitigate CVE-2020-11023
• contains: HIVE-24074 Incorrect handling of timestamp in Parquet/Avro when written in certain time zones in versions before Hive 3.x
• contains: HIVE-24113 NPE in GenericUDFToUnixTimeStamp
• contains: HIVE-24322 In case of direct insert, the attempt ID has to be checked when reading the manifest files
• contains: HIVE-24584 IndexOutOfBoundsException from Kryo when running msck repair
• contains: HIVE-24694 Early connection close to release server resources during creating
• contains: HIVE-24786 JDBC HttpClient should retry for idempotent and unsent http methods
• contains: HIVE-24999 HiveSubQueryRemoveRule generates invalid plan for IN subquery with multiple correlations
• contains: HIVE-25104 Backward incompatible timestamp serialization in Parquet for certain timezones
• contains: HIVE-25839 Upgrade Log4j2 to 2.17.1 due to CVE-2021-44832
• contains: HIVE-26127 INSERT OVERWRITE throws FileNotFound when destination partition is deleted
• contains: HIVE-26300 Upgrade Jackson databind version to 2.12.6.1+ to avoid CVE-2020-36518
• contains: HIVE-25093 date_format() UDF is returning output in UTC time zone only
• contains: HIVE-21857 Sort conditions in a filter predicate to accelerate query processing
• contains: HIVE-26948 Backport HIVE-21456 to branch-3
• contains: HIVE-26949 Backport HIVE-26071 to branch-3
• contains: HIVE-27001 Backport HIVE-26633 to branch-3
• contains: HIVE-26702 Backport HIVE-17317 (DBCP and HikariCP property configuration support) to 3.2.0.
• contains: HIVE-27136 Backport HIVE-27129 to branch-3
• contains: HIVE-27261 Backport HIVE-27258 to branch-3
• contains: HIVE-23363 Upgrade DataNucleus dependency to 5.2
• contains: HIVE-21171 Skip creating scratch dirs for tez if RPC is on
• contains: HIVE-21338 Remove order by and limit for aggregates
• contains: HIVE-21815 Stats in ORC file are parsed twice
• contains: HIVE-21924 Split text files even if header/footer exists
• contains: HIVE-22106 PCR: Remove cross-query synchronization for the partition-eval
• contains: HIVE-22221 Llap external client - Need to reduce LlapBaseInputFormat#getSplits() footprint
• contains: HIVE-22827 Update Flatbuffer version
• contains: HIVE-26173 Upgrade derby to 10.14.2.0
• contains: HIVE-12371 Adding a timeout connection parameter for JDBC
• contains: HIVE-21457 Perf optimizations in ORC split-generation
• contains: HIVE-25516 ITestDbTxnManager is broken after HIVE-24120
• contains: HIVE-21991 Upgrade ORC version to 1.5.6
• contains: HIVE-22182 SemanticAnalyzer populates map which is not used at all
• contains: HIVE-25697 Upgrade commons-compress to 1.21
• contains: HIVE-25920 Bump Xerce2 to 2.12.2
• contains: HIVE-26078 Upgrade gson to 2.8.9
• contains: HIVE-26080 Upgrade accumulo-core to 1.10.1
• contains: HIVE-26081 Upgrade ant to 1.10.9
• contains: HIVE-26412 Create interface to fetch available slots during split calculation
• contains: HIVE-25098 [CVE-2020-13949] Upgrade thrift from 0.13.0 to 0.14.1
• contains: HIVE-25945 Upgrade H2 database version to 2.1.210
• is a child of: HIVE-26748 Prepare for Hive 3.2.0 Release
• is related to: HIVE-26743 backport HIVE-24694 to 3.1.x
• links to: GitHub Pull Request #4521

Sub-Tasks

• Backport HIVE-23619 to branch-3: Resolved
• Backport of HIVE-22599 to branch-3: Resolved
• Backport of HIVE-22121 to branch-3: Resolved
• Backport of HIVE-22136 and HIVE-22227 : Turn on tez.bucket.pruning: Resolved
• Backport of HIVE-18786 "NPE in Hive windowing functions": Resolved
• Backport of HIVE-24020 : Fix Automatic compaction for streaming ingest with dynamic partition: Resolved
• Backport of HIVE-21717 : Rename is failing for directory in move task.: Resolved
• Backport of HIVE-23262 : Remove dependency on activemq: Resolved
• Backport of HIVE-24964: Backport HIVE-22453: Describe table unnecessarily fetches partitions: Open
• Backport of HIVE-24803: WorkloadManager doesn't update allocation and metrics after Kill Trigger action: Resolved
• Backport of HIVE-19133: HS2 WebUI phase-wise performance metrics not showing correctly: Resolved
• Backport of HIVE-22566: Drop table involved in materialized view leaves the table in inconsistent state: Resolved
• Backport of HIVE-24751: Kill trigger in workload manager fails with "No privilege" exception when authorization is disabled.: Open
• Backport of HIVE-22099: Several date related UDFs can't handle Julian dates properly since HIVE-20007: Resolved
• Backport of HIVE-23046: Separate housekeeping thread from initiator flag: Resolved
• Backport of HIVE-23058: Compaction task reattempt fails with FileAlreadyExistsException: Resolved
• Backport of HIVE-24882: Compaction task reattempt fails with FileAlreadyExistsException for DeleteEventWriter: Open
• Backport HIVE-22170 (unix timestamp) to branch-3: Open
• Backport of HIVE-19416, HIVE-20293, HIVE-20117, HIVE-18453, HIVE-20833, HIVE-21498: Resolved
• Backport of HIVE-24201: Resolved
• Fix orc_merge10.q test in branch-3: Resolved
• Backport of HIVE-21924, HIVE-22769, HIVE-24224, HIVE-24381,HIVE-21905: Resolved
• Backport of HIVE-25659: Metastore direct sql queries with IN/(NOT IN) should be split based on max parameters allowed by SQL DB: Resolved
• Backport of HIVE-24039 : Update jquery version to mitigate CVE-2020-11023 (#1403): Open
• Backport of HIVE-22120 - Fix wrong results/ArrayOutOfBound exception in left outer map joins on specific boundary conditions: Open
• Backport of HIVE-22273: Access check is failed when a temporary directory is removed: Resolved
• Backport of HIVE-15956: StackOverflowError when drop lots of partitions: Open
• Backport of HIVE-22219: Bringing a node manager down blocks restart of LLAP service: Resolved
• Backport of HIVE-22275: OperationManager.queryIdOperation does not properly clean up multiple queryIds: Resolved
• Backport of HIVE-22219: Bringing a node manager down blocks restart of LLAP service: Open
• Backport of HIVE-22113: Prevent LLAP shutdown on AMReporter related RuntimeException: Resolved
• Backport of HIVE-22208 : Column name with reserved keyword is unescaped when query including join on table with mask column is re-written: Resolved
• Backport of HIVE-22360, HIVE-20619 : MultiDelimitSerDe returns wrong results in last column when the loaded file has more columns than those in table schema: Resolved
• Backport HIVE-22094 : queries failing with ClassCastException: hive.ql.exec.vector.DecimalColumnVector cannot be cast to hive.ql.exec.vector.Decimal64ColumnVector: Open
• Backport of HIVE-22405: Add ColumnVector support for ProlepticCalendar: Resolved
• Backport of HIVE-21815: Stats in ORC file are parsed twice: Resolved
• Backport of HIVE-18702: INSERT OVERWRITE TABLE doesn't clean the table directory before overwriting: Resolved
• Backport of HIVE-21296: Dropping varchar partition throw exception: Resolved
• Backport of HIVE-21799: NullPointerException in DynamicPartitionPruningOptimization, when join key is on aggregation column: Resolved
• Backport of HIVE-22661: Compaction fails on non bucketed table with data loaded inpath: Open
• Backport of HIVE-21001: Upgrade to calcite-1.19: Open
• Backport of HIVE-25299: Open
• Backport of HIVE-21924: Split text files even if header/footer exists: Resolved
• Backport of HIVE-21915: Hive with TEZ UNION ALL and UDTF results in data loss: Resolved
• Backport of HIVE-22498: Resolved
• HIVE-21167 : Bucketing: Bucketing version 1 is incorrectly partitioning data: Resolved
• Backport of HIVE-19661 : switch Hive UDFs to use Re2J regex engine: Resolved
• Backport of HIVE-21171: Skip creating scratch dirs for tez if RPC is on: Resolved
• Backport of HIVE-21182 Skip setting up hive scratch dir during planning: Resolved
• Backport of HIVE-22106: Remove cross-query synchronization for the partition-eval: Resolved
• Backport of HIVE-22115: Prevent the creation of query routing appender if property is set to false: Resolved
• Backport of HIVE-22161 UDF: FunctionRegistry synchronizes on org.apache.hadoop.hive.ql.udf.UDFType class: Resolved
• Backport of HIVE-22168: Remove very expensive logging from the llap cache hotpath: Resolved
• Backport of HIVE-22169: Tez: SplitGenerator tries to look for plan files which won't exist for Tez: Resolved
• Backport of HIVE-22204: Beeline option to show/not show execution report: Resolved
• Backport of HIVE-21009: Adding ability for user to set bind user: Resolved
• Backport of HIVE-21280 : Null pointer exception on running compaction against a MM table.: Resolved
• Backport of HIVE-20619: Include MultiDelimitSerDe in HiveServer2 By Default: Resolved
• Backport of HIVE-18284: Fix NPE when inserting data with 'distribute by' clause with dynpart sort optimization: Resolved
• Backport of HIVE-25446: Wrong execption thrown if capacity<=0: Resolved
• Backport of HIVE-12371: Adding a timeout connection parameter for JDBC: Resolved
• Backport of HIVE-16812: VectorizedOrcAcidRowBatchReader doesn't filter delete events: Open
• Backport of HIVE-17917, HIVE-21457, HIVE-22582: Resolved
• Backport of HIVE-22903 : Vectorized row_number() resets the row number after one batch in case of constant expression in partition clause: Resolved
• Backport of HIVE-21126: Allow session level queries in LlapBaseInputFormat#getSplits() before actual get_splits() call: Resolved
• Backport of HIVE-22398: Remove legacy code that can cause issue with new Yarn releases: Resolved
• Backport of HIVE-25458: Unix_timestamp() with string input give wrong result: Open
• Backport of HIVE-19081: Add partition should prevent loading acid files: Open
• Backport of HIVE-24693 : Convert timestamps to zoned times without string operations: Open
• Backport of HIVE-22841: Open
• Backport of HIVE-25299 : HIVE-21104: PTF with nested structure throws ClassCastException: Resolved
• Backport of HIVE-22595: Dynamic partition inserts fail on Avro table table with external schema: Open
• Backport of HIVE-20462: "CREATE VIEW IF NOT EXISTS" fails if view already exists: Open
• Backport of HIVE-25235: Remove ThreadPoolExecutorWithOomHook: Resolved
• Backport of HIVE-23396: Many fixes and improvements to stabilize tests: Resolved
• Added org.bouncycastle as dependency and excluded the dependencies of the same which lead to version mismatch while running tests: Resolved
• Backward compatibility for Hive with Components like Spark: Open
• Backport of HIVE-20397: HiveStrictManagedMigration updates: Open
• Backport of HIVE-20052, HIVE-20093, HIVE-20203, HIVE-20290, HIVE-20300, HIVE-20312, HIVE-20044, HIVE-21966: Resolved
• Backport of HIVE-20364, HIVE-20549 to branch-3: Resolved
• Backport of HIVE-20467, HIVE-20508, HIVE-20550: Resolved
• Fix explainanalyze_2.q test: Open
• Backport of HIVE-20536, HIVE-20632, HIVE-20511, HIVE-20560, HIVE-20631, HIVE-20637, HIVE-20609, HIVE-20439: Resolved
• Backport of HIVE-20629, HIVE-20705, HIVE-20734: Resolved
• Backport of HIVE-20082 ,22170 ,22331 , 24113 , 25458 , 25577: Open
• Backport of HIVE-21715: Adding a new partition specified by location (which is empty) leads to Exceptions: Open
• Backport of HIVE-22593: Compaction fails on non bucketed table with data loaded inpath: Open
• Backport of HIVE-23023: MR compaction ignores column schema evolution: Open
• Backport of HIVE-24138: Llap external client flow is broken due to netty shading: Open
• Backport of HIVE-25535: Control cleaning obsolete directories/files of a table via property: Open
• Backport of HIVE-20430, HIVE-20613, HIVE-20615: Open
• Backport of HIVE-21032, HIVE-20590, HIVE-20409, HIVE-20919: Open
• Backport of HIVE-22336: Updates should be pushed to the Metastore backend DB before creating the notification event: Open
• Backport of HIVE-21241, HIVE-22400, HIVE-24693, HIVE-25306: Open
• Backport HIVE-25250: Fix TestHS2ImpersonationWithRemoteMS.testImpersonation: Open
• Backport of HIVE-25093 and HIVE-25268: Open
• Backport of HIVE-22840: Race condition in formatters of TimestampColumnVector and DateColumnVector: Open
• Backport of HIVE-20437: Handle schema evolution from Float, Double and Decimal.: Open
• Backport of HIVE-21729: Arrow serializer sometimes shifts timestamp by one second: Open
• Backport of HIVE-21862: ORC ppd produces wrong result with timestamp: Open
• Backport HIVE-21866: LLAP status service driver may get stuck with wrong Yarn app ID: Resolved
• Backport of HIVE-24693, HIVE-20433, HIVE-20792, HIVE-20838, HIVE-21241, HIVE-22099, HIVE-25093, HIVE-22400, HIVE-25306, HIVE-25268to branch-3: Resolved
• Backport of HIVE-27324: Hive query with NOT IN condition is giving incorrect results when the sub query table contains the null value.: Open
• Support DISTINCT with GROUP BY: Open
• Backport of HIVE-22429, HIVE-14898, HIVE-22231, HIVE-20507, HIVE-24786 to branch-3: Resolved
• Backport of HIVE-22373, HIVE-25553, HIVE-23561, HIVE-24321, HIVE-22856, HIVE-22973, HIVE-21729, HIVE-23022, HIVE-23034, HIVE-22221, HIVE-23230 to branch-3: Resolved
• Backport of HIVE-21862, HIVE-20437, HIVE-22589, HIVE-22840, HIVE-24074, HIVE-25104 to branch-3: Resolved
• Backport of HIVE-24808, HIVE-24811, HIVE-22476, HIVE-25299, HIVE-22200, HIVE-21544, HIVE-22841, HIVE-20292, HIVE-17040, HIVE-20718, HIVE-20193 to branch-3: Resolved
• Backport of HIVE-20082, HIVE-19564, HIVE-20174, HIVE-20197, HIVE-20207, HIVE-20245, HIVE-20294, HIVE-26447, HIVE-20315, HIVE-22170, HIVE-22331, HIVE-24113, HIVE-25458, HIVE-25577, HIVE-25403: Resolved