[CRUNCH-659] Upgrade to Hive 2.x

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.0.0
• Assignee: Stephen Durfey
• Reporter: Stephen Durfey
• Created: 07/Nov/17 23:22
• Updated: 20/Nov/19 14:20
• Resolved: 08/Dec/17 16:27

Description

I've been working on CRUNCH-340 to finish implementing the HCatSource and HCatTarget. It seems to be in a better place now that crunch only supports hadoop 2. I was looking to target as high of a version of hive/hcat as possible with minimal impact on the code base and dependencies.

Hive 2.3.1 is out now. That relies upon hadoop 2.7.2, but HBase doesn't bump up to that version until HBase 2.x. Trying to run with hadoop 2.7.2 causes test failures in crunch-hbase. I'm not sure if that is going to cause runtime issues as the minicluster wouldn't even start due to a package name change in hadoop-hdfs (for the class StorageType) that's causing a no class found error.

Hive 2.1.0 relies upon Hadoop 2.6.0, and that plays nice with HBase 1.x. However, the class StructField (inside TupleObjectInspector for ORC files) has a new abstract method added to it that would need to be implemented that was introduced after 2.x of Hive. Other than that everything runs fine.

Currently Crunch is on 0.13.1 of Hive, so it's pretty far behind. I'm just kind of looking for feedback on the version bumps that should be targeted for my changes in CRUNCH-340. I wanted to take care of those first in a separate JIRA before introducing new code against a higher Hive version.

Issue Links

• relates to: CRUNCH-340 Create HCatSource and HCatTarget