Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-14269 Performance optimizations for data on S3
  3. HIVE-16295

Add support for using Hadoop's S3A OutputCommitter

          Details

          • Type: Sub-task
          • Status: Patch Available
          • Priority: Major
          • Resolution: Unresolved
          • Affects Version/s: None
          • Fix Version/s: None
          • Component/s: None
          • Labels:
            None
          • Target Version/s:

            Description

            Hive doesn't have integration with Hadoop's OutputCommitter, it uses a NullOutputCommitter and uses its own commit logic spread across FileSinkOperator, MoveTask, and Hive.

            The Hadoop community is building an OutputCommitter that integrates with S3Guard and does a safe, coordinate commit of data on S3 inside individual tasks (HADOOP-13786). If Hive can integrate with this new OutputCommitter there would be a lot of benefits to Hive-on-S3:

            • Data is only written once; directly committing data at a task level means no renames are necessary
            • The commit is done safely, in a coordinated manner; duplicate tasks (from task retries or speculative execution) should not step on each other

              Attachments

              1. HIVE-16295.1.WIP.patch
                44 kB
                Sahil Takiar
              2. HIVE-16295.2.WIP.patch
                44 kB
                Sahil Takiar
              3. HIVE-16295.3.WIP.patch
                45 kB
                Sahil Takiar
              4. HIVE-16295.4.patch
                110 kB
                Sahil Takiar
              5. HIVE-16295.5.patch
                118 kB
                Sahil Takiar
              6. HIVE-16295.6.patch
                124 kB
                Sahil Takiar
              7. HIVE-16295.7.patch
                124 kB
                Sahil Takiar
              8. HIVE-16295.8.patch
                133 kB
                Sahil Takiar

                Issue Links

                is blocked by

                Bug - A problem which impairs or prevents the functions of the product. HIVE-19217 Upgrade to Hadoop 3.1.0

                • Major - Major loss of function.
                • Resolved

                Task - A task that needs to be done. HIVE-18319 Upgrade to Hadoop 3.0.0

                • Major - Major loss of function.
                • Resolved
                is related to

                Sub-task - The sub-task of the issue HADOOP-15421 Stabilise/formalise the JSON _SUCCESS format used in the S3A committers

                • Major - Major loss of function.
                • Open

                New Feature - A new feature of the product, which has yet to be developed. HADOOP-13786 Add S3A committers for zero-rename commits to S3 endpoints

                • Major - Major loss of function.
                • Resolved
                relates to

                Sub-task - The sub-task of the issue HIVE-19321 Dynamic Partitioning Integration with Hadoop's S3A OutputCommitter

                • Major - Major loss of function.
                • Open
                links to

                Web Link RB

                  Activity

                    People

                    • Assignee:
                      stakiar Sahil Takiar
                      Reporter:
                      stakiar Sahil Takiar
                    • Votes:
                      0 Vote for this issue
                      Watchers:
                      10 Start watching this issue

                      Dates

                      • Created:
                        Updated: