Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-3131

enabling BLOCK compression for map outputs breaks the reduce progress counters

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.17.0, 0.17.1, 0.17.2, 0.18.0, 0.19.0
    • Fix Version/s: 0.19.0
    • Component/s: None
    • Labels:
      None

      Description

      Enabling map output compression and setting the compression type to BLOCK causes the progress counters during the reduce to go crazy and report progress counts over 100%.

      This is problematic for speculative execution because it thinks the tasks are doing fine.

        Attachments

        1. HADOOP-3131-v2.patch
          9 kB
          Matei Zaharia
        2. HADOOP-3131-v3.patch
          9 kB
          Matei Zaharia
        3. HADOOP-3131-v4.patch
          13 kB
          Matei Zaharia
        4. HADOOP-3131-v5.patch
          13 kB
          Matei Zaharia
        5. HADOOP-3131-v6.patch
          14 kB
          Matei Zaharia
        6. merge-progress.patch
          5 kB
          Matei Zaharia
        7. merge-progress-trunk.patch
          9 kB
          Matei Zaharia
        8. Picture 1.png
          106 kB
          Colin Evans

          Activity

            People

            • Assignee:
              matei@eecs.berkeley.edu Matei Zaharia
              Reporter:
              colinhevans Colin Evans
            • Votes:
              1 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: