Details
-
Sub-task
-
Status: Resolved
-
Major
-
Resolution: Won't Fix
-
None
-
None
-
None
-
None
Description
If hive.merge.mapfiles is True, and the output table to write is on S3, then Hive will generate a conditional plan where smaller files will be merged into larger sizes.
If the output files written by the initial MR job are small, then a second MR job is run to merge the output into larger files (a copy from S3 to S3 in the current code).
If the original output files are large enough, then the conditional task is followed by a move/rename which is very expensive in S3.
We should keep staging data on HDFS previous to copying them to S3 as final files.