Pig
  1. Pig
  2. PIG-179

On hadoop 0.16, some jobs using combiner fail with an NPE

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.0.0
    • Fix Version/s: 0.1.0
    • Component/s: None
    • Labels:
      None
    • Environment:

      Hadoop 0.16

      Description

      Some jobs (it appears to only be larger jobs) now fail with an NPE in the combiner code on this line:

      PigSplit split = PigInputFormat.PigRecordReader.getPigRecordReader().getPigFileSplit();
      

      Looking into the PigRecordReader a comment in the class indicates that, as implemented, it depends on the mapper and splitter (and in this case the combiner as well) running in the same thread. It seems that in some cases in hadoop 0.16 this is no longer the case.

        Activity

        Alan Gates created issue -
        Alan Gates made changes -
        Field Original Value New Value
        Attachment PIG-179.patch [ 12379191 ]
        Alan Gates made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Alan Gates made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Fix Version/s 0.1.0 [ 12312848 ]
        Resolution Fixed [ 1 ]
        Alan Gates made changes -
        Resolution Fixed [ 1 ]
        Status Resolved [ 5 ] Reopened [ 4 ]
        Alan Gates made changes -
        Resolution Fixed [ 1 ]
        Status Reopened [ 4 ] Resolved [ 5 ]
        Alan Gates made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            Alan Gates
            Reporter:
            Alan Gates
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development