HBase
  1. HBase
  2. HBASE-11335

Fix the TABLE_DIR param in TableSnapshotInputFormat

    Details

      Description

      In class TableSnapshotInputFormat or TableSnapshotInputFormatImpl
      in the function

      public static void setInput(Job job, String snapshotName, Path restoreDir) throws IOException {
      

      we are setting restoreDir (temporary root) to tableDir

      conf.set(TABLE_DIR_KEY, restoreDir.toString());
      

      The above parameter is used to get the InputSplits, especially for
      calculating favorable hosts in the function

      Path tableDir = new Path(conf.get(TABLE_DIR_KEY));
      
      List<String> hosts = getBestLocations(conf,
                HRegion.computeHDFSBlocksDistribution(conf, htd, hri, tableDir));
      

      This will lead to returning a empty HDFSBlocksDistribution, as there is
      will be no directory with name as the region name from hri in the restored
      root directory, which will lead to scheduling of non local tasks.

      The change is simple in the sense, is to call the

      FSUtils.getTableDir(rootDir, tableDesc.getTableName()) 

      in the getSplits function

      more discussion in the comments below

      https://issues.apache.org/jira/browse/HBASE-8369?focusedCommentId=14012085&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14012085

      1. HBASE_11335-0.96-v1.patch
        3 kB
        deepankar
      2. HBASE_11335-trunk-v1.patch
        2 kB
        deepankar

        Activity

        Hide
        deepankar added a comment -

        patches are attached, the patch for trunk and 0.98 are same, for 0.96 the file is different
        and also I checked that this issue is not there for 0.94, as they are already passing the correct
        string as parameter

        Show
        deepankar added a comment - patches are attached, the patch for trunk and 0.98 are same, for 0.96 the file is different and also I checked that this issue is not there for 0.94, as they are already passing the correct string as parameter
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12649958/HBASE_11335-trunk-v1.patch
        against trunk revision .
        ATTACHMENT ID: 12649958

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 lineLengths. The patch does not introduce lines longer than 100

        +1 site. The mvn site goal succeeds with this patch.

        +1 core tests. The patch passed unit tests in .

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/9754//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9754//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9754//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9754//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9754//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9754//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9754//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9754//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9754//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9754//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/9754//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12649958/HBASE_11335-trunk-v1.patch against trunk revision . ATTACHMENT ID: 12649958 +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 +1 site . The mvn site goal succeeds with this patch. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/9754//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9754//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9754//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9754//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9754//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9754//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9754//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9754//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9754//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9754//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/9754//console This message is automatically generated.
        Hide
        deepankar added a comment -

        ping Enis Soztutar, Can you take a look, we want to put this in our production

        Show
        deepankar added a comment - ping Enis Soztutar , Can you take a look, we want to put this in our production
        Hide
        Enis Soztutar added a comment -

        I've committed this to master. Waiting for Andrew Purtell's go ahead for 0.98.

        Show
        Enis Soztutar added a comment - I've committed this to master. Waiting for Andrew Purtell 's go ahead for 0.98.
        Hide
        Enis Soztutar added a comment -

        Thanks deepankar for the patch.

        Show
        Enis Soztutar added a comment - Thanks deepankar for the patch.
        Hide
        Andrew Purtell added a comment -

        +1

        Show
        Andrew Purtell added a comment - +1
        Hide
        Enis Soztutar added a comment -

        Committed to 0.98 and 0.96 (bug fix) as well.

        Show
        Enis Soztutar added a comment - Committed to 0.98 and 0.96 (bug fix) as well.
        Hide
        deepankar added a comment -

        Thanks for your help

        Show
        deepankar added a comment - Thanks for your help
        Hide
        Hudson added a comment -

        FAILURE: Integrated in hbase-0.96 #406 (See https://builds.apache.org/job/hbase-0.96/406/)
        HBASE-11335 Fix the TABLE_DIR param in TableSnapshotInputFormat (deepankar) (enis: rev c6abf4f0a93a447bc8d0b4c926f7bba523069cad)

        • hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableSnapshotInputFormat.java
        Show
        Hudson added a comment - FAILURE: Integrated in hbase-0.96 #406 (See https://builds.apache.org/job/hbase-0.96/406/ ) HBASE-11335 Fix the TABLE_DIR param in TableSnapshotInputFormat (deepankar) (enis: rev c6abf4f0a93a447bc8d0b4c926f7bba523069cad) hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableSnapshotInputFormat.java
        Hide
        Hudson added a comment -

        FAILURE: Integrated in hbase-0.96-hadoop2 #280 (See https://builds.apache.org/job/hbase-0.96-hadoop2/280/)
        HBASE-11335 Fix the TABLE_DIR param in TableSnapshotInputFormat (deepankar) (enis: rev c6abf4f0a93a447bc8d0b4c926f7bba523069cad)

        • hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableSnapshotInputFormat.java
        Show
        Hudson added a comment - FAILURE: Integrated in hbase-0.96-hadoop2 #280 (See https://builds.apache.org/job/hbase-0.96-hadoop2/280/ ) HBASE-11335 Fix the TABLE_DIR param in TableSnapshotInputFormat (deepankar) (enis: rev c6abf4f0a93a447bc8d0b4c926f7bba523069cad) hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableSnapshotInputFormat.java
        Hide
        Hudson added a comment -

        ABORTED: Integrated in HBase-0.98-on-Hadoop-1.1 #347 (See https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/347/)
        HBASE-11335 Fix the TABLE_DIR param in TableSnapshotInputFormat (deepankar) (enis: rev e2ae542a05adad3bd23d4cfae4d878a2a2374052)

        • hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableSnapshotInputFormatImpl.java
        Show
        Hudson added a comment - ABORTED: Integrated in HBase-0.98-on-Hadoop-1.1 #347 (See https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/347/ ) HBASE-11335 Fix the TABLE_DIR param in TableSnapshotInputFormat (deepankar) (enis: rev e2ae542a05adad3bd23d4cfae4d878a2a2374052) hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableSnapshotInputFormatImpl.java
        Hide
        Hudson added a comment -

        ABORTED: Integrated in HBase-0.98 #367 (See https://builds.apache.org/job/HBase-0.98/367/)
        HBASE-11335 Fix the TABLE_DIR param in TableSnapshotInputFormat (deepankar) (enis: rev e2ae542a05adad3bd23d4cfae4d878a2a2374052)

        • hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableSnapshotInputFormatImpl.java
        Show
        Hudson added a comment - ABORTED: Integrated in HBase-0.98 #367 (See https://builds.apache.org/job/HBase-0.98/367/ ) HBASE-11335 Fix the TABLE_DIR param in TableSnapshotInputFormat (deepankar) (enis: rev e2ae542a05adad3bd23d4cfae4d878a2a2374052) hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableSnapshotInputFormatImpl.java
        Hide
        Hudson added a comment -

        ABORTED: Integrated in HBase-TRUNK #5251 (See https://builds.apache.org/job/HBase-TRUNK/5251/)
        HBASE-11335 Fix the TABLE_DIR param in TableSnapshotInputFormat (deepankar) (enis: rev 5af264c5b5d97643abd2142bfde51fe83f967453)

        • hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableSnapshotInputFormatImpl.java
        Show
        Hudson added a comment - ABORTED: Integrated in HBase-TRUNK #5251 (See https://builds.apache.org/job/HBase-TRUNK/5251/ ) HBASE-11335 Fix the TABLE_DIR param in TableSnapshotInputFormat (deepankar) (enis: rev 5af264c5b5d97643abd2142bfde51fe83f967453) hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableSnapshotInputFormatImpl.java
        Hide
        Enis Soztutar added a comment -

        Closing this issue after 0.99.0 release.

        Show
        Enis Soztutar added a comment - Closing this issue after 0.99.0 release.

          People

          • Assignee:
            Unassigned
            Reporter:
            deepankar
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development