[HIVE-14269] Performance optimizations for data on S3

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.1.0
• Assignee: Sergio Peña
• Reporter: Sergio Peña
• Created: 18/Jul/16 19:23
• Updated: 15/Feb/19 21:36

Description

Working with tables that resides on Amazon S3 (or any other object store) have several performance impact when reading or writing data, and also consistency issues.

This JIRA is an umbrella task to monitor all the performance improvements that can be done in Hive to work better with S3 data.

Issue Links

• depends upon: HADOOP-11694 Über-jira: S3a phase II: robustness, scale and performance
• depends upon: HIVE-14323 Reduce number of FS permissions and redundant FS operations
• is depended upon by: HADOOP-13525 Optimize uses of FS operations in the ASF analysis frameworks and libraries
• is related to: HIVE-16277 Exchange Partition between filesystems throws "IllegalArgumentException Wrong FS"
• is related to: HIVE-1620 Patch to write directly to S3 from Hive
• relates to: HADOOP-13204 Über-jira: S3a phase III: scale and tuning
• relates to: HIVE-14920 S3: Optimize SimpleFetchOptimizer::checkThreshold()

Sub-Tasks

• Write temporary data to HDFS when doing inserts on tables located on S3: Resolved
• Skip 'distcp' call when copying data from HDSF to S3: Patch Available
• FileSinkOperator should not rename files to final paths when S3 is the default destination: Reopened
• ConditionalResolverMergeFiles should keep staging data on HDFS, then copy (no rename) to S3: Resolved
• Remove Hive file listing during split computation: Closed
• Change .hive-staging directory created on S3 to HDFS when using INSERT OVERWRITE: Resolved
• Add integration tests for hive on S3: Resolved
• S3-to-S3 Renames: Files should be moved individually rather than at a directory level: Resolved
• Remove extra MoveTask operators from the ConditionalTask: Resolved
• Last MR job in Hive should be able to write to a different scratch directory: Reopened
• Investigate if staging data on S3 can always go under the scratch dir: Open
• Optimize Utilities.getInputPaths() so each listStatus of a partition is done in parallel: Resolved
• Blobstores should use fs.listFiles(path, recursive=true) rather than FileUtils.listStatusRecursively: Resolved
• Fix NoSuchMethodError: com.amazonaws.services.s3.transfer.TransferManagerConfiguration.setMultipartUploadThreshold(I)V: Closed
• Add support for using Hadoop's S3A OutputCommitter: Patch Available
• inheritPerms should be conditional based on the target filesystem: Resolved
• druid-hdfs-storage is pulling in hadoop-aws-2.7.x and aws SDK, creating classpath problems on hadoop 3.x: Closed
• Dynamic Partitioning Integration with Hadoop's S3A OutputCommitter: Open
• Ability to selectively run tests in TestBlobstoreCliDriver: Open
• PerfLogger integration for critical Hive-on-S3 paths: Closed
• Add --SORT_QUERY_RESULTS to hive-blobstore/map_join.q.out: Closed
• Add Parquet specific tests to BlobstoreCliDriver: Closed