[HIVE-14269] Performance optimizations for data on S3

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.1.0
• Assignee: Sergio Peña
• Reporter: Sergio Peña
• Created: 18/Jul/16 19:23
• Updated: 15/Feb/19 21:36

Description

Working with tables that resides on Amazon S3 (or any other object store) have several performance impact when reading or writing data, and also consistency issues.

This JIRA is an umbrella task to monitor all the performance improvements that can be done in Hive to work better with S3 data.

Issue Links

• depends upon: HADOOP-11694 Über-jira: S3a phase II: robustness, scale and performance
• depends upon: HIVE-14323 Reduce number of FS permissions and redundant FS operations
• is depended upon by: HADOOP-13525 Optimize uses of FS operations in the ASF analysis frameworks and libraries
• is related to: HIVE-16277 Exchange Partition between filesystems throws "IllegalArgumentException Wrong FS"
• is related to: HIVE-1620 Patch to write directly to S3 from Hive
• relates to: HADOOP-13204 Über-jira: S3a phase III: scale and tuning
• relates to: HIVE-14920 S3: Optimize SimpleFetchOptimizer::checkThreshold()

Sub-Tasks

• Skip 'distcp' call when copying data from HDSF to S3: Patch Available
• FileSinkOperator should not rename files to final paths when S3 is the default destination: Reopened
• Last MR job in Hive should be able to write to a different scratch directory: Reopened
• Investigate if staging data on S3 can always go under the scratch dir: Open
• Add support for using Hadoop's S3A OutputCommitter: Patch Available
• Dynamic Partitioning Integration with Hadoop's S3A OutputCommitter: Open
• Ability to selectively run tests in TestBlobstoreCliDriver: Open