Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-20113

Shuffle avoidance: Disable 1-1 edges for sorted shuffle

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.0.0
    • Component/s: Tez

      Description

      The sorted shuffle avoidance can have some issues when the shuffle data gets broken up into multiple chunks on disk.

      The 1-1 edge cannot skip the tez final merge - there's no reason for 1-1 to have a final merge at all, it should open a single compressed file and write a single index entry.

      Until the shuffle issue is resolved & a lot more testing, it is prudent to disable the optimization for sorted shuffle edges and stop rewriting the RS(sorted) = = = RS(sorted) into RS(sorted) = = = RS(FORWARD).

        Attachments

        1. HIVE-20113.1.patch
          1 kB
          Gopal Vijayaraghavan
        2. HIVE-20113.10.patch
          352 kB
          Vineet Garg
        3. HIVE-20113.10.patch
          352 kB
          Vineet Garg
        4. HIVE-20113.2.patch
          71 kB
          Gopal Vijayaraghavan
        5. HIVE-20113.3.patch
          73 kB
          Ashutosh Chauhan
        6. HIVE-20113.4.patch
          143 kB
          Gopal Vijayaraghavan
        7. HIVE-20113.4.patch
          143 kB
          Ashutosh Chauhan
        8. HIVE-20113.5.patch
          85 kB
          Vineet Garg
        9. HIVE-20113.6.patch
          352 kB
          Vineet Garg
        10. HIVE-20113.7.patch
          352 kB
          Vineet Garg
        11. HIVE-20113.8.patch
          352 kB
          Vineet Garg
        12. HIVE-20113.9.patch
          352 kB
          Vineet Garg

          Activity

            People

            • Assignee:
              vgarg Vineet Garg
              Reporter:
              gopalv Gopal Vijayaraghavan
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: