HBase
  1. HBase
  2. HBASE-5480

Fixups to MultithreadedTableMapper for Hadoop 0.23.2+

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.94.0, 0.95.0
    • Component/s: mapreduce
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      There are two issues:

      • StatusReporter has a new method getProgress()
      • Mapper and reducer context objects can no longer be directly instantiated.

      See attached patch. I'm not thrilled with the added reflection but it was the minimally intrusive change.

      Raised the priority to critical because compilation fails.

      1. HBASE-5480.patch
        2 kB
        Andrew Purtell

        Issue Links

          Activity

          Hide
          Lars Hofhansl added a comment -

          Committed to 0.94 as well.

          Show
          Lars Hofhansl added a comment - Committed to 0.94 as well.
          Hide
          Lars Hofhansl added a comment -

          Reopening, so I won't forget about it.

          Show
          Lars Hofhansl added a comment - Reopening, so I won't forget about it.
          Hide
          stack added a comment -

          But this should go into 0.94, no?

          Yes.

          Show
          stack added a comment - But this should go into 0.94, no? Yes.
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK #2674 (See https://builds.apache.org/job/HBase-TRUNK/2674/)
          HBASE-5480. Fixups to MultithreadedTableMapper for Hadoop 0.23 (Revision 1298259)

          Result = FAILURE
          apurtell :
          Files :

          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/MultithreadedTableMapper.java
          Show
          Hudson added a comment - Integrated in HBase-TRUNK #2674 (See https://builds.apache.org/job/HBase-TRUNK/2674/ ) HBASE-5480 . Fixups to MultithreadedTableMapper for Hadoop 0.23 (Revision 1298259) Result = FAILURE apurtell : Files : /hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/MultithreadedTableMapper.java
          Hide
          Lars Hofhansl added a comment -

          But this should go into 0.94, no?

          Show
          Lars Hofhansl added a comment - But this should go into 0.94, no?
          Hide
          stack added a comment -

          I don't have this class in 0.92 Andrew, so 0.92 should be fine (Thanks for flagging it)

          Show
          stack added a comment - I don't have this class in 0.92 Andrew, so 0.92 should be fine (Thanks for flagging it)
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK-security #131 (See https://builds.apache.org/job/HBase-TRUNK-security/131/)
          HBASE-5480. Fixups to MultithreadedTableMapper for Hadoop 0.23 (Revision 1298259)

          Result = SUCCESS
          apurtell :
          Files :

          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/MultithreadedTableMapper.java
          Show
          Hudson added a comment - Integrated in HBase-TRUNK-security #131 (See https://builds.apache.org/job/HBase-TRUNK-security/131/ ) HBASE-5480 . Fixups to MultithreadedTableMapper for Hadoop 0.23 (Revision 1298259) Result = SUCCESS apurtell : Files : /hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/MultithreadedTableMapper.java
          Hide
          Lars Hofhansl added a comment -

          I'm seeing the commit in trunk but not 0.94.

          Show
          Lars Hofhansl added a comment - I'm seeing the commit in trunk but not 0.94.
          Hide
          Andrew Purtell added a comment -

          Passes TestMulitthreadedTableMapper with 0.23 and default Hadoop profiles.

          @Stack: Into 0.92 branch too? (You open it yet?)

          Show
          Andrew Purtell added a comment - Passes TestMulitthreadedTableMapper with 0.23 and default Hadoop profiles. @Stack: Into 0.92 branch too? (You open it yet?)
          Hide
          Ted Yu added a comment -

          Do you want to commit the patch Andy ?

          Show
          Ted Yu added a comment - Do you want to commit the patch Andy ?
          Hide
          Andrew Purtell added a comment -

          There was a constructor called at that site already, and another constructor called by reflection already above it. This only adds a small incremental cost.

          Show
          Andrew Purtell added a comment - There was a constructor called at that site already, and another constructor called by reflection already above it. This only adds a small incremental cost.
          Hide
          stack added a comment -

          +1 Looks grand Andy. Reflection is per map invocation? So, per row? I suppose in scheme of things not too bad.

          Show
          stack added a comment - +1 Looks grand Andy. Reflection is per map invocation? So, per row? I suppose in scheme of things not too bad.
          Hide
          Andrew Purtell added a comment -

          Corrected patch with --no-prefix

          Show
          Andrew Purtell added a comment - Corrected patch with --no-prefix

            People

            • Assignee:
              Andrew Purtell
              Reporter:
              Andrew Purtell
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development