[DRILL-6540] Upgrade to HADOOP-3.0 libraries

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.14.0
• Fix Version/s: 1.17.0
• Component/s: Tools, Build & Test
• Labels: ready-to-commit
• Assignee: Anton Gozhiy
• Reporter: Vitalii Diravka
• Reviewer: Vova Vysotskyi
• Created: 26/Jun/18 09:54
• Updated: 16/Dec/19 15:29
• Resolved: 04/Dec/19 12:59

Description

Currently Drill uses 2.7.4 version of hadoop libraries (hadoop-common, hadoop-hdfs, hadoop-annotations, hadoop-aws, hadoop-yarn-api, hadoop-client, hadoop-yarn-client).
A year ago the Hadoop 3.0 was released and recently it was updated to Hadoop 3.2.0.

To use Drill under Hadoop3.0 distribution we need this upgrade. Also the newer version includes new features, which can be useful for Drill.
This upgrade is also needed to leverage the newest version of Zookeeper libraries and Hive 3.1 version.

Issue Links

• is a parent of: DRILL-7051 Upgrade to Jetty 9.3
• is blocked by: HADOOP-15749 Unhandled Log4JLogger explicit reference breaks non-log4j users
• is depended upon by: DRILL-6558 Drill query fails when file name contains semicolon
• is related to: DRILL-5733 Unable to SELECT from parquet file with Hadoop 2.7.4
• is related to: DRILL-4977 Reading parquet metadata cache from S3 with fadvise=random and Hadoop 3 generates a large number of requests
• is related to: DRILL-6521 When using Hadoop configs, allow dfs connection to be unset
• is related to: DRILL-6580 Upgrade Curator libraries
• is related to: DRILL-6534 Upgrade ZooKeeper patch version to 3.4.12 and add Apache Curator to dependencyManagement
• is required by: DRILL-6532 Support of Archive.org as an S3 Provider
• is required by: DRILL-6604 Upgrade Drill Hive client to Hive3.1 version
• links to: GitHub Pull Request #1732
• links to: GitHub Pull Request #1895