Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-3131

enabling BLOCK compression for map outputs breaks the reduce progress counters

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 0.17.0, 0.17.1, 0.17.2, 0.18.0, 0.19.0
    • 0.19.0
    • None
    • None

    Description

      Enabling map output compression and setting the compression type to BLOCK causes the progress counters during the reduce to go crazy and report progress counts over 100%.

      This is problematic for speculative execution because it thinks the tasks are doing fine.

      Attachments

        1. Picture 1.png
          106 kB
          Colin Evans
        2. merge-progress.patch
          5 kB
          Matei Zaharia
        3. merge-progress-trunk.patch
          9 kB
          Matei Zaharia
        4. HADOOP-3131-v2.patch
          9 kB
          Matei Zaharia
        5. HADOOP-3131-v3.patch
          9 kB
          Matei Zaharia
        6. HADOOP-3131-v4.patch
          13 kB
          Matei Zaharia
        7. HADOOP-3131-v5.patch
          13 kB
          Matei Zaharia
        8. HADOOP-3131-v6.patch
          14 kB
          Matei Zaharia

        Activity

          People

            matei@eecs.berkeley.edu Matei Zaharia
            colinhevans Colin Evans
            Votes:
            1 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: