HBase
  1. HBase
  2. HBASE-10546

Two scanner objects are open for each hbase map task but only one scanner object is closed

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.96.2, 0.98.1, 0.99.0, 0.94.17
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Map reduce framework calls createRecordReader of the TableInputFormat/MultiTableInputFormat to get the record reader instance. In this method, we are initializing the TableRecordReaderImpl (restart method). This initializes the scanner object. After this, map reduce framework calls initialize on the RecordReader. In our case, this calls restart of the TableRecordReaderImpl again. Here, it doesn't close the first scanner. At the end of the task, only the second scanner object is closed. Because of this, the smallest read point of HRegion is affected.

      We don't need to initialize the RecordReader in the createRecordReader method and we need to close the scanner object in the restart method. (incase if the restart method is called because of exceptions in the nextKeyValue method)

      1. trunk-HBASE-10546.patch
        0.9 kB
        Vasu Mariyala
      2. 0.94-HBASE-10546.patch
        2 kB
        Vasu Mariyala

        Activity

        Hide
        Ted Yu added a comment -

        +1

        Show
        Ted Yu added a comment - +1
        Hide
        Lars Hofhansl added a comment -

        Looks good. +1

        Show
        Lars Hofhansl added a comment - Looks good. +1
        Hide
        Lars Hofhansl added a comment -

        For 0.94, would it be safer to leave the initialize calls in place? I find it hard looking at the code that this will not cause any issues if somebody subclassed any of the involved classes. Seems safer to just have the change in restart (in the worst case we create a close a scanner per input split).

        Show
        Lars Hofhansl added a comment - For 0.94, would it be safer to leave the initialize calls in place? I find it hard looking at the code that this will not cause any issues if somebody subclassed any of the involved classes. Seems safer to just have the change in restart (in the worst case we create a close a scanner per input split).
        Hide
        Lars Hofhansl added a comment -

        Also see HBASE-6069, which added the initialize we're removing now (to avoid an NPE with Hive).
        With that in mind, we should not remove the initialize() calls.

        Let's just fix restart(...)

        Show
        Lars Hofhansl added a comment - Also see HBASE-6069 , which added the initialize we're removing now (to avoid an NPE with Hive). With that in mind, we should not remove the initialize() calls. Let's just fix restart(...)
        Hide
        Lars Hofhansl added a comment -

        Unless there are objections I'll commit to trunk version of the patch to all branches. We can always go back later and remove spurious calls to initialize. Right now I'd prefer to keep them (Vasu Mariyala).

        Show
        Lars Hofhansl added a comment - Unless there are objections I'll commit to trunk version of the patch to all branches. We can always go back later and remove spurious calls to initialize. Right now I'd prefer to keep them ( Vasu Mariyala ).
        Hide
        Lars Hofhansl added a comment -

        Committed trunk version to all branches.
        Thanks Vasu Mariyala.

        Show
        Lars Hofhansl added a comment - Committed trunk version to all branches. Thanks Vasu Mariyala .
        Hide
        Hudson added a comment -

        FAILURE: Integrated in HBase-0.94-security #417 (See https://builds.apache.org/job/HBase-0.94-security/417/)
        HBASE-10546 Two scanner objects are open for each hbase map task but only one scanner object is closed. (Vasu Mariyala) (larsh: rev 1569487)

        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/mapreduce/TableRecordReaderImpl.java
        Show
        Hudson added a comment - FAILURE: Integrated in HBase-0.94-security #417 (See https://builds.apache.org/job/HBase-0.94-security/417/ ) HBASE-10546 Two scanner objects are open for each hbase map task but only one scanner object is closed. (Vasu Mariyala) (larsh: rev 1569487) /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/mapreduce/TableRecordReaderImpl.java
        Hide
        Hudson added a comment -

        FAILURE: Integrated in HBase-0.94-on-Hadoop-2 #29 (See https://builds.apache.org/job/HBase-0.94-on-Hadoop-2/29/)
        HBASE-10546 Two scanner objects are open for each hbase map task but only one scanner object is closed. (Vasu Mariyala) (larsh: rev 1569487)

        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/mapreduce/TableRecordReaderImpl.java
        Show
        Hudson added a comment - FAILURE: Integrated in HBase-0.94-on-Hadoop-2 #29 (See https://builds.apache.org/job/HBase-0.94-on-Hadoop-2/29/ ) HBASE-10546 Two scanner objects are open for each hbase map task but only one scanner object is closed. (Vasu Mariyala) (larsh: rev 1569487) /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/mapreduce/TableRecordReaderImpl.java
        Hide
        Hudson added a comment -

        FAILURE: Integrated in HBase-0.94-JDK7 #58 (See https://builds.apache.org/job/HBase-0.94-JDK7/58/)
        HBASE-10546 Two scanner objects are open for each hbase map task but only one scanner object is closed. (Vasu Mariyala) (larsh: rev 1569487)

        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/mapreduce/TableRecordReaderImpl.java
        Show
        Hudson added a comment - FAILURE: Integrated in HBase-0.94-JDK7 #58 (See https://builds.apache.org/job/HBase-0.94-JDK7/58/ ) HBASE-10546 Two scanner objects are open for each hbase map task but only one scanner object is closed. (Vasu Mariyala) (larsh: rev 1569487) /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/mapreduce/TableRecordReaderImpl.java
        Hide
        Hudson added a comment -

        FAILURE: Integrated in HBase-0.94 #1294 (See https://builds.apache.org/job/HBase-0.94/1294/)
        HBASE-10546 Two scanner objects are open for each hbase map task but only one scanner object is closed. (Vasu Mariyala) (larsh: rev 1569487)

        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/mapreduce/TableRecordReaderImpl.java
        Show
        Hudson added a comment - FAILURE: Integrated in HBase-0.94 #1294 (See https://builds.apache.org/job/HBase-0.94/1294/ ) HBASE-10546 Two scanner objects are open for each hbase map task but only one scanner object is closed. (Vasu Mariyala) (larsh: rev 1569487) /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/mapreduce/TableRecordReaderImpl.java
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in HBase-0.98 #165 (See https://builds.apache.org/job/HBase-0.98/165/)
        HBASE-10546 Two scanner objects are open for each hbase map task but only one scanner object is closed. (Vasu Mariyala) (larsh: rev 1569495)

        • /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableRecordReaderImpl.java
        Show
        Hudson added a comment - SUCCESS: Integrated in HBase-0.98 #165 (See https://builds.apache.org/job/HBase-0.98/165/ ) HBASE-10546 Two scanner objects are open for each hbase map task but only one scanner object is closed. (Vasu Mariyala) (larsh: rev 1569495) /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableRecordReaderImpl.java
        Hide
        Hudson added a comment -

        FAILURE: Integrated in hbase-0.96 #299 (See https://builds.apache.org/job/hbase-0.96/299/)
        HBASE-10546 Two scanner objects are open for each hbase map task but only one scanner object is closed. (Vasu Mariyala) (larsh: rev 1569493)

        • /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableRecordReaderImpl.java
        Show
        Hudson added a comment - FAILURE: Integrated in hbase-0.96 #299 (See https://builds.apache.org/job/hbase-0.96/299/ ) HBASE-10546 Two scanner objects are open for each hbase map task but only one scanner object is closed. (Vasu Mariyala) (larsh: rev 1569493) /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableRecordReaderImpl.java
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in HBase-0.98-on-Hadoop-1.1 #154 (See https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/154/)
        HBASE-10546 Two scanner objects are open for each hbase map task but only one scanner object is closed. (Vasu Mariyala) (larsh: rev 1569495)

        • /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableRecordReaderImpl.java
        Show
        Hudson added a comment - SUCCESS: Integrated in HBase-0.98-on-Hadoop-1.1 #154 (See https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/154/ ) HBASE-10546 Two scanner objects are open for each hbase map task but only one scanner object is closed. (Vasu Mariyala) (larsh: rev 1569495) /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableRecordReaderImpl.java
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in HBase-TRUNK #4928 (See https://builds.apache.org/job/HBase-TRUNK/4928/)
        HBASE-10546 Two scanner objects are open for each hbase map task but only one scanner object is closed. (Vasu Mariyala) (larsh: rev 1569500)

        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableRecordReaderImpl.java
        Show
        Hudson added a comment - SUCCESS: Integrated in HBase-TRUNK #4928 (See https://builds.apache.org/job/HBase-TRUNK/4928/ ) HBASE-10546 Two scanner objects are open for each hbase map task but only one scanner object is closed. (Vasu Mariyala) (larsh: rev 1569500) /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableRecordReaderImpl.java
        Hide
        Hudson added a comment -

        FAILURE: Integrated in hbase-0.96-hadoop2 #207 (See https://builds.apache.org/job/hbase-0.96-hadoop2/207/)
        HBASE-10546 Two scanner objects are open for each hbase map task but only one scanner object is closed. (Vasu Mariyala) (larsh: rev 1569493)

        • /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableRecordReaderImpl.java
        Show
        Hudson added a comment - FAILURE: Integrated in hbase-0.96-hadoop2 #207 (See https://builds.apache.org/job/hbase-0.96-hadoop2/207/ ) HBASE-10546 Two scanner objects are open for each hbase map task but only one scanner object is closed. (Vasu Mariyala) (larsh: rev 1569493) /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableRecordReaderImpl.java
        Hide
        Hudson added a comment -

        FAILURE: Integrated in HBase-TRUNK-on-Hadoop-1.1 #93 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-1.1/93/)
        HBASE-10546 Two scanner objects are open for each hbase map task but only one scanner object is closed. (Vasu Mariyala) (larsh: rev 1569500)

        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableRecordReaderImpl.java
        Show
        Hudson added a comment - FAILURE: Integrated in HBase-TRUNK-on-Hadoop-1.1 #93 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-1.1/93/ ) HBASE-10546 Two scanner objects are open for each hbase map task but only one scanner object is closed. (Vasu Mariyala) (larsh: rev 1569500) /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableRecordReaderImpl.java

          People

          • Assignee:
            Vasu Mariyala
            Reporter:
            Vasu Mariyala
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development