[HIVE-18116] Hive + HDFS EC Supportability and Testing Improvements

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Target Version/s: 3.0.0
• Assignee: Andrew Sherman
• Reporter: Sahil Takiar
• Created: 21/Nov/17 16:47
• Updated: 11/Dec/18 21:33
• Resolved: 11/Dec/18 21:33

Description

Now that we are on Hadoop 3.x, we can start integrating with HDFS Erasure Coding (see https://hadoop.apache.org/docs/r3.0.0-alpha2/hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html for details).

First step is to add some tests using a custom CliDriver - we can do something similar to what we did for encryption.

Next step will be some supportability improvements - like printing out in the explain plan when a query is reading a EC file.

Sub-Tasks

• Create TestCliDriver for HDFS EC: Closed
• Explain Extended should indicate if a file being read is an EC file: Closed
• show partitions should say whether a partition is stored via EC: Resolved
• Set hadoop.version=3.1.0 in standalone-metastore: Closed
• Add logging of runtime statistics indicating when Hdfs Erasure Coding is used by MR: Closed
• Add logging of runtime statistics indicating when Hdfs Erasure Coding is used by Spark: Open