Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-20113

Shuffle avoidance: Disable 1-1 edges for sorted shuffle

Log workAgile BoardRank to TopRank to BottomVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 4.0.0
    • Tez

    Description

      The sorted shuffle avoidance can have some issues when the shuffle data gets broken up into multiple chunks on disk.

      The 1-1 edge cannot skip the tez final merge - there's no reason for 1-1 to have a final merge at all, it should open a single compressed file and write a single index entry.

      Until the shuffle issue is resolved & a lot more testing, it is prudent to disable the optimization for sorted shuffle edges and stop rewriting the RS(sorted) = = = RS(sorted) into RS(sorted) = = = RS(FORWARD).

      Attachments

        1. HIVE-20113.1.patch
          1 kB
          Gopal Vijayaraghavan
        2. HIVE-20113.2.patch
          71 kB
          Gopal Vijayaraghavan
        3. HIVE-20113.3.patch
          73 kB
          Ashutosh Chauhan
        4. HIVE-20113.4.patch
          143 kB
          Ashutosh Chauhan
        5. HIVE-20113.4.patch
          143 kB
          Gopal Vijayaraghavan
        6. HIVE-20113.5.patch
          85 kB
          Vineet Garg
        7. HIVE-20113.6.patch
          352 kB
          Vineet Garg
        8. HIVE-20113.7.patch
          352 kB
          Vineet Garg
        9. HIVE-20113.8.patch
          352 kB
          Vineet Garg
        10. HIVE-20113.9.patch
          352 kB
          Vineet Garg
        11. HIVE-20113.10.patch
          352 kB
          Vineet Garg
        12. HIVE-20113.10.patch
          352 kB
          Vineet Garg

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            vgarg Vineet Garg Assign to me
            gopalv Gopal Vijayaraghavan
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Issue deployment