Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-3993

Graceful handling of codec errors during decompression

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.1, 1.0.2
    • Fix Version/s: 1.2.0, 2.0.2-alpha
    • Component/s: mrv1, mrv2
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      When using a compression codec for intermediate compression, some cases of corrupt data can cause the codec to throw exceptions other than IOException (eg java.lang.InternalError). This will currently cause the whole reduce task to fail, instead of simply treating it like another case of a failed fetch.

      1. MR-3993_branch1.patch
        9 kB
        Karthik Kambatla
      2. MR-3993_trunk.patch
        6 kB
        Karthik Kambatla
      3. MR-3993_branch1.patch
        5 kB
        Karthik Kambatla
      4. MR-3993_trunk.patch
        4 kB
        Karthik Kambatla

        Activity

        Gavin made changes -
        Assignee Karthik Kambatla [ kkambatl ] Karthik Kambatla [ kasha ]
        Arun C Murthy made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Arun C Murthy made changes -
        Fix Version/s 2.0.2-alpha [ 12322471 ]
        Fix Version/s 2.1.0-alpha [ 12321442 ]
        Matt Foley made changes -
        Target Version/s 1.1.0, 2.0.0-alpha [ 12317960, 12320354 ] 1.2.0, 2.1.0-alpha [ 12321661, 12321442 ]
        Alejandro Abdelnur made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hadoop Flags Reviewed [ 10343 ]
        Target Version/s 1.1.0, 2.0.0-alpha [ 12317960, 12320354 ] 2.0.0-alpha, 1.1.0 [ 12320354, 12317960 ]
        Fix Version/s 1.2.0 [ 12321661 ]
        Fix Version/s 2.0.1-alpha [ 12321442 ]
        Resolution Fixed [ 1 ]
        Karthik Kambatla (Inactive) made changes -
        Attachment MR-3993_branch1.patch [ 12535718 ]
        Attachment MR-3993_trunk.patch [ 12535719 ]
        Karthik Kambatla (Inactive) made changes -
        Summary reduce fetch catch clause should catch RTEs as well Graceful handling of codec errors during decompression
        Target Version/s 1.1.0, 2.0.0-alpha [ 12317960, 12320354 ] 2.0.0-alpha, 1.1.0 [ 12320354, 12317960 ]
        Karthik Kambatla (Inactive) made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Target Version/s 1.1.0, 2.0.0-alpha [ 12317960, 12320354 ] 2.0.0-alpha, 1.1.0 [ 12320354, 12317960 ]
        Karthik Kambatla (Inactive) made changes -
        Attachment MR-3993_branch1.patch [ 12535407 ]
        Karthik Kambatla (Inactive) made changes -
        Attachment MR-3993_branch1.patch [ 12535471 ]
        Attachment MR-3993_trunk.patch [ 12535472 ]
        Karthik Kambatla (Inactive) made changes -
        Attachment MR-3993_branch1.patch [ 12535407 ]
        Karthik Kambatla (Inactive) made changes -
        Attachment MR-3993_branch1.patch [ 12535284 ]
        Karthik Kambatla (Inactive) made changes -
        Attachment MR-3993_branch1.patch [ 12535242 ]
        Karthik Kambatla (Inactive) made changes -
        Attachment MR-3993_branch1.patch [ 12535284 ]
        Karthik Kambatla (Inactive) made changes -
        Attachment MR-3993_branch1.patch [ 12535242 ]
        Karthik Kambatla (Inactive) made changes -
        Assignee Karthik Kambatla [ kkambatl ]
        Matt Foley made changes -
        Target Version/s 2.0.0, 1.0.3 [ 12320354, 12320250 ] 1.1.0, 2.0.0 [ 12317960, 12320354 ]
        Robert Joseph Evans made changes -
        Target Version/s 1.0.3, 0.23.3 [ 12320250, 12320060 ] 1.0.3, 2.0.0 [ 12320250, 12320354 ]
        Matt Foley made changes -
        Field Original Value New Value
        Target Version/s 0.23.3, 1.0.2 [ 12320060, 12320047 ] 0.23.3, 1.0.3 [ 12320060, 12320250 ]
        Todd Lipcon created issue -

          People

          • Assignee:
            Karthik Kambatla
            Reporter:
            Todd Lipcon
          • Votes:
            0 Vote for this issue
            Watchers:
            15 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development