HBase
  1. HBase
  2. HBASE-10622

Improve log and Exceptions in Export Snapshot

    Details

      Description

      from the logs of export snapshot is not really clear what's going on,
      adding some extra information useful to debug, and in some places the real exception can be thrown

      1. HBASE-10622-v0.patch
        16 kB
        Matteo Bertozzi
      2. HBASE-10622-v1.patch
        17 kB
        Matteo Bertozzi
      3. HBASE-10622-v2.patch
        17 kB
        Matteo Bertozzi
      4. HBASE-10622-v3.patch
        17 kB
        Matteo Bertozzi
      5. HBASE-10622-v4.patch
        18 kB
        Matteo Bertozzi

        Issue Links

          Activity

          Hide
          Hudson added a comment -

          SUCCESS: Integrated in HBase-0.98 #200 (See https://builds.apache.org/job/HBase-0.98/200/)
          HBASE-10622 Improve log and Exceptions in Export Snapshot (mbertozzi: rev 1574032)

          • /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshot.java
          Show
          Hudson added a comment - SUCCESS: Integrated in HBase-0.98 #200 (See https://builds.apache.org/job/HBase-0.98/200/ ) HBASE-10622 Improve log and Exceptions in Export Snapshot (mbertozzi: rev 1574032) /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshot.java
          Hide
          Hudson added a comment -

          FAILURE: Integrated in HBase-TRUNK-on-Hadoop-1.1 #107 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-1.1/107/)
          HBASE-10622 Improve log and Exceptions in Export Snapshot (mbertozzi: rev 1574031)

          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshot.java
          Show
          Hudson added a comment - FAILURE: Integrated in HBase-TRUNK-on-Hadoop-1.1 #107 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-1.1/107/ ) HBASE-10622 Improve log and Exceptions in Export Snapshot (mbertozzi: rev 1574031) /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshot.java
          Hide
          Hudson added a comment -

          FAILURE: Integrated in HBase-0.94-JDK7 #72 (See https://builds.apache.org/job/HBase-0.94-JDK7/72/)
          HBASE-10622 Improve log and Exceptions in Export Snapshot (mbertozzi: rev 1574034)

          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshot.java
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshot.java
          Show
          Hudson added a comment - FAILURE: Integrated in HBase-0.94-JDK7 #72 (See https://builds.apache.org/job/HBase-0.94-JDK7/72/ ) HBASE-10622 Improve log and Exceptions in Export Snapshot (mbertozzi: rev 1574034) /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshot.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshot.java
          Hide
          Hudson added a comment -

          FAILURE: Integrated in HBase-0.98-on-Hadoop-1.1 #186 (See https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/186/)
          HBASE-10622 Improve log and Exceptions in Export Snapshot (mbertozzi: rev 1574032)

          • /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshot.java
          Show
          Hudson added a comment - FAILURE: Integrated in HBase-0.98-on-Hadoop-1.1 #186 (See https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/186/ ) HBASE-10622 Improve log and Exceptions in Export Snapshot (mbertozzi: rev 1574032) /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshot.java
          Hide
          Hudson added a comment -

          FAILURE: Integrated in HBase-0.94 #1308 (See https://builds.apache.org/job/HBase-0.94/1308/)
          HBASE-10622 Improve log and Exceptions in Export Snapshot (mbertozzi: rev 1574034)

          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshot.java
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshot.java
          Show
          Hudson added a comment - FAILURE: Integrated in HBase-0.94 #1308 (See https://builds.apache.org/job/HBase-0.94/1308/ ) HBASE-10622 Improve log and Exceptions in Export Snapshot (mbertozzi: rev 1574034) /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshot.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshot.java
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in hbase-0.96 #326 (See https://builds.apache.org/job/hbase-0.96/326/)
          HBASE-10622 Improve log and Exceptions in Export Snapshot (mbertozzi: rev 1574033)

          • /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshot.java
          Show
          Hudson added a comment - SUCCESS: Integrated in hbase-0.96 #326 (See https://builds.apache.org/job/hbase-0.96/326/ ) HBASE-10622 Improve log and Exceptions in Export Snapshot (mbertozzi: rev 1574033) /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshot.java
          Hide
          Hudson added a comment -

          FAILURE: Integrated in HBase-TRUNK #4976 (See https://builds.apache.org/job/HBase-TRUNK/4976/)
          HBASE-10622 Improve log and Exceptions in Export Snapshot (mbertozzi: rev 1574031)

          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshot.java
          Show
          Hudson added a comment - FAILURE: Integrated in HBase-TRUNK #4976 (See https://builds.apache.org/job/HBase-TRUNK/4976/ ) HBASE-10622 Improve log and Exceptions in Export Snapshot (mbertozzi: rev 1574031) /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshot.java
          Hide
          Hudson added a comment -

          FAILURE: Integrated in HBase-0.94-on-Hadoop-2 #39 (See https://builds.apache.org/job/HBase-0.94-on-Hadoop-2/39/)
          HBASE-10622 Improve log and Exceptions in Export Snapshot (mbertozzi: rev 1574034)

          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshot.java
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshot.java
          Show
          Hudson added a comment - FAILURE: Integrated in HBase-0.94-on-Hadoop-2 #39 (See https://builds.apache.org/job/HBase-0.94-on-Hadoop-2/39/ ) HBASE-10622 Improve log and Exceptions in Export Snapshot (mbertozzi: rev 1574034) /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshot.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshot.java
          Hide
          Hudson added a comment -

          FAILURE: Integrated in HBase-0.94-security #429 (See https://builds.apache.org/job/HBase-0.94-security/429/)
          HBASE-10622 Improve log and Exceptions in Export Snapshot (mbertozzi: rev 1574034)

          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshot.java
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshot.java
          Show
          Hudson added a comment - FAILURE: Integrated in HBase-0.94-security #429 (See https://builds.apache.org/job/HBase-0.94-security/429/ ) HBASE-10622 Improve log and Exceptions in Export Snapshot (mbertozzi: rev 1574034) /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshot.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshot.java
          Hide
          Hudson added a comment -

          FAILURE: Integrated in hbase-0.96-hadoop2 #224 (See https://builds.apache.org/job/hbase-0.96-hadoop2/224/)
          HBASE-10622 Improve log and Exceptions in Export Snapshot (mbertozzi: rev 1574033)

          • /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshot.java
          Show
          Hudson added a comment - FAILURE: Integrated in hbase-0.96-hadoop2 #224 (See https://builds.apache.org/job/hbase-0.96-hadoop2/224/ ) HBASE-10622 Improve log and Exceptions in Export Snapshot (mbertozzi: rev 1574033) /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshot.java
          Hide
          Jerry He added a comment -

          other jira, it requires a new InputFormat/RecordReader with the progress based on the file size and not on the number of lines in the input file. The only progress that we track is the current file copy

          Agree. Not an easy thing to do. It seems that DistCp has the same issue.

          Looks good. Thanks.

          Show
          Jerry He added a comment - other jira, it requires a new InputFormat/RecordReader with the progress based on the file size and not on the number of lines in the input file. The only progress that we track is the current file copy Agree. Not an easy thing to do. It seems that DistCp has the same issue. Looks good. Thanks.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12632257/HBASE-10622-v4.patch
          against trunk revision .
          ATTACHMENT ID: 12632257

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.

          +1 hadoop1.1. The patch compiles against the hadoop 1.1 profile.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 lineLengths. The patch does not introduce lines longer than 100

          +1 site. The mvn site goal succeeds with this patch.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hbase.client.TestHCM

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/8864//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8864//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8864//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8864//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8864//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8864//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8864//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8864//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8864//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8864//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/8864//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12632257/HBASE-10622-v4.patch against trunk revision . ATTACHMENT ID: 12632257 +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 hadoop1.0 . The patch compiles against the hadoop 1.0 profile. +1 hadoop1.1 . The patch compiles against the hadoop 1.1 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. -1 findbugs . The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 +1 site . The mvn site goal succeeds with this patch. -1 core tests . The patch failed these unit tests: org.apache.hadoop.hbase.client.TestHCM Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/8864//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8864//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8864//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8864//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8864//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8864//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8864//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8864//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8864//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8864//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/8864//console This message is automatically generated.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12632218/HBASE-10622-v4.patch
          against trunk revision .
          ATTACHMENT ID: 12632218

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.

          +1 hadoop1.1. The patch compiles against the hadoop 1.1 profile.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 lineLengths. The patch does not introduce lines longer than 100

          +1 site. The mvn site goal succeeds with this patch.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hbase.snapshot.TestSecureExportSnapshot
          org.apache.hadoop.hbase.snapshot.TestExportSnapshot

          -1 core zombie tests. There are 1 zombie test(s):

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/8862//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8862//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8862//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8862//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8862//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8862//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8862//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8862//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8862//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8862//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/8862//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12632218/HBASE-10622-v4.patch against trunk revision . ATTACHMENT ID: 12632218 +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 hadoop1.0 . The patch compiles against the hadoop 1.0 profile. +1 hadoop1.1 . The patch compiles against the hadoop 1.1 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. -1 findbugs . The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 +1 site . The mvn site goal succeeds with this patch. -1 core tests . The patch failed these unit tests: org.apache.hadoop.hbase.snapshot.TestSecureExportSnapshot org.apache.hadoop.hbase.snapshot.TestExportSnapshot -1 core zombie tests . There are 1 zombie test(s): Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/8862//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8862//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8862//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8862//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8862//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8862//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8862//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8862//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8862//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8862//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/8862//console This message is automatically generated.
          Hide
          Matteo Bertozzi added a comment -

          In the run(), can we cleanup/delete snapshotTmpDir if Step 2 failed so that we don't ask the user to manually clean it since it comes from our Step 1 copy?

          There is an -overwrite option that does already this, but I think the general problem should be solved by the last line of this patch where the snapshotTmpDir is removed if you get an exception.

          Another issue is probably more involved, and does not need to be covered in this JIRA. It is the overall progress reporting of the ExportSnapshot job.

          other jira, it requires a new InputFormat/RecordReader with the progress based on the file size and not on the number of lines in the input file. The only progress that we track is the current file copy

          Show
          Matteo Bertozzi added a comment - In the run(), can we cleanup/delete snapshotTmpDir if Step 2 failed so that we don't ask the user to manually clean it since it comes from our Step 1 copy? There is an -overwrite option that does already this, but I think the general problem should be solved by the last line of this patch where the snapshotTmpDir is removed if you get an exception. Another issue is probably more involved, and does not need to be covered in this JIRA. It is the overall progress reporting of the ExportSnapshot job. other jira, it requires a new InputFormat/RecordReader with the progress based on the file size and not on the number of lines in the input file. The only progress that we track is the current file copy
          Hide
          Jerry He added a comment -

          A few more comments (since you are doing the improvement ...)

          -        // Verify that the written size match
          -        if (totalBytesWritten != inputFileSize) {
          -          String msg = "number of bytes copied not matching copied=" + totalBytesWritten +
          -                       " expected=" + inputFileSize + " for file=" + inputPath;
          -          throw new IOException(msg);
          

          You think this is unnecessary?

          In the run(), can we cleanup/delete snapshotTmpDir if Step 2 failed so that we don't ask the user to manually clean it since it comes from our Step 1 copy?

          Can we add a job counter say 'COPIES_FILES' to be along side with 'BYTES_COPIED'?

          Another issue is probably more involved, and does not need to be covered in this JIRA. It is the overall progress reporting of the ExportSnapshot job.
          For example,
          hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot snapshot1 -copy-to /user/biadmin/mysnapshots -mappers 30

          14/03/02 12:19:54 INFO mapred.JobClient:  map 0% reduce 0%
          14/03/02 12:20:12 INFO mapred.JobClient:  map 6% reduce 0%
          14/03/02 12:20:13 INFO mapred.JobClient:  map 44% reduce 0%
          14/03/02 12:20:19 INFO mapred.JobClient:  map 83% reduce 0%
          

          There is about 130G to export. But it takes just a few secs to get to 83%, after the first around of mappers are launched, and will stay there for a long time.
          Similarly at the end it will show 100% for a long time while there are mappers still running.
          he map progress percentage is quite inaccurate with regard to the over progress.

          Show
          Jerry He added a comment - A few more comments (since you are doing the improvement ...) - // Verify that the written size match - if (totalBytesWritten != inputFileSize) { - String msg = "number of bytes copied not matching copied=" + totalBytesWritten + - " expected=" + inputFileSize + " for file=" + inputPath; - throw new IOException(msg); You think this is unnecessary? In the run(), can we cleanup/delete snapshotTmpDir if Step 2 failed so that we don't ask the user to manually clean it since it comes from our Step 1 copy? Can we add a job counter say 'COPIES_FILES' to be along side with 'BYTES_COPIED'? Another issue is probably more involved, and does not need to be covered in this JIRA. It is the overall progress reporting of the ExportSnapshot job. For example, hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot snapshot1 -copy-to /user/biadmin/mysnapshots -mappers 30 14/03/02 12:19:54 INFO mapred.JobClient: map 0% reduce 0% 14/03/02 12:20:12 INFO mapred.JobClient: map 6% reduce 0% 14/03/02 12:20:13 INFO mapred.JobClient: map 44% reduce 0% 14/03/02 12:20:19 INFO mapred.JobClient: map 83% reduce 0% There is about 130G to export. But it takes just a few secs to get to 83%, after the first around of mappers are launched, and will stay there for a long time. Similarly at the end it will show 100% for a long time while there are mappers still running. he map progress percentage is quite inaccurate with regard to the over progress.
          Hide
          Jonathan Hsieh added a comment -

          lgtm. is findbugs you or something else?

          Show
          Jonathan Hsieh added a comment - lgtm. is findbugs you or something else?
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12631831/HBASE-10622-v3.patch
          against trunk revision .
          ATTACHMENT ID: 12631831

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.

          +1 hadoop1.1. The patch compiles against the hadoop 1.1 profile.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 lineLengths. The patch does not introduce lines longer than 100

          +1 site. The mvn site goal succeeds with this patch.

          +1 core tests. The patch passed unit tests in .

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/8857//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8857//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8857//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8857//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8857//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8857//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8857//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8857//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8857//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8857//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/8857//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12631831/HBASE-10622-v3.patch against trunk revision . ATTACHMENT ID: 12631831 +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 hadoop1.0 . The patch compiles against the hadoop 1.0 profile. +1 hadoop1.1 . The patch compiles against the hadoop 1.1 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. -1 findbugs . The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 +1 site . The mvn site goal succeeds with this patch. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/8857//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8857//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8857//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8857//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8857//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8857//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8857//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8857//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8857//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8857//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/8857//console This message is automatically generated.
          Hide
          Matteo Bertozzi added a comment -

          Jerry He thanks for the catch, I forgot that that last part of the string was done on purpose...
          btw now the result looks like this, from the TT page:
          copied 52.2 M/410.1 M (12.7%) from ...
          copied 410.1 M/410.1 M (100.0%) from ...

          Show
          Matteo Bertozzi added a comment - Jerry He thanks for the catch, I forgot that that last part of the string was done on purpose... btw now the result looks like this, from the TT page: copied 52.2 M/410.1 M (12.7%) from ... copied 410.1 M/410.1 M (100.0%) from ...
          Hide
          Jerry He added a comment -

          Are you reversing HBASE-9060 in the code by putting the path inthe

          Are you reversing HBASE-9060 in the code by putting the path in the format?

          Show
          Jerry He added a comment - Are you reversing HBASE-9060 in the code by putting the path inthe Are you reversing HBASE-9060 in the code by putting the path in the format?
          Hide
          Jerry He added a comment -

          Hi, Matteo

          Are you reversing HBASE-9060 in the code by putting the path inthe

          Also, do you want to put a '%' charactor after:
          (totalBytesWritten/(float)inputFileSize) * 100.0f

          Show
          Jerry He added a comment - Hi, Matteo Are you reversing HBASE-9060 in the code by putting the path inthe Also, do you want to put a '%' charactor after: (totalBytesWritten/(float)inputFileSize) * 100.0f
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12631733/HBASE-10622-v2.patch
          against trunk revision .
          ATTACHMENT ID: 12631733

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.

          +1 hadoop1.1. The patch compiles against the hadoop 1.1 profile.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 lineLengths. The patch does not introduce lines longer than 100

          +1 site. The mvn site goal succeeds with this patch.

          +1 core tests. The patch passed unit tests in .

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/8852//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8852//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8852//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8852//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8852//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8852//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8852//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8852//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8852//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8852//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/8852//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12631733/HBASE-10622-v2.patch against trunk revision . ATTACHMENT ID: 12631733 +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 hadoop1.0 . The patch compiles against the hadoop 1.0 profile. +1 hadoop1.1 . The patch compiles against the hadoop 1.1 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. -1 findbugs . The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 +1 site . The mvn site goal succeeds with this patch. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/8852//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8852//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8852//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8852//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8852//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8852//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8852//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8852//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8852//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8852//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/8852//console This message is automatically generated.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12631707/HBASE-10622-v1.patch
          against trunk revision .
          ATTACHMENT ID: 12631707

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.

          +1 hadoop1.1. The patch compiles against the hadoop 1.1 profile.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 lineLengths. The patch does not introduce lines longer than 100

          +1 site. The mvn site goal succeeds with this patch.

          +1 core tests. The patch passed unit tests in .

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/8849//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8849//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8849//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8849//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8849//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8849//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8849//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8849//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8849//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8849//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/8849//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12631707/HBASE-10622-v1.patch against trunk revision . ATTACHMENT ID: 12631707 +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 hadoop1.0 . The patch compiles against the hadoop 1.0 profile. +1 hadoop1.1 . The patch compiles against the hadoop 1.1 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. -1 findbugs . The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 +1 site . The mvn site goal succeeds with this patch. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/8849//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8849//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8849//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8849//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8849//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8849//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8849//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8849//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8849//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8849//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/8849//console This message is automatically generated.
          Hide
          Matteo Bertozzi added a comment -

          job.getStatus().getFailureInfo() is not present in all the version, so when you end up getting this.
          I've added a TODO, and I'll skip it for now

          java.lang.NoSuchMethodError: org.apache.hadoop.mapreduce.Job.getStatus()Lorg/apache/hadoop/mapreduce/JobStatus
          
          Show
          Matteo Bertozzi added a comment - job.getStatus().getFailureInfo() is not present in all the version, so when you end up getting this. I've added a TODO, and I'll skip it for now java.lang.NoSuchMethodError: org.apache.hadoop.mapreduce.Job.getStatus()Lorg/apache/hadoop/mapreduce/JobStatus
          Hide
          Matteo Bertozzi added a comment -
          It will help to use job.getStatus().getFailureInfo() if the copy job failed?

          We already get each mapper stack trace in the "main" log if something fail,
          but I guess is not bad if we report it again at the end. let me add that

          Another area is to somehow intelligently to estimate the number of copy mappers needed based on the size and number of files? Similar to DistCp?
          

          sure, I was thinking to rewrite the file grouping later to keep in consideration also the locations..
          but I guess adding a default number of mappers here is short enough

          Show
          Matteo Bertozzi added a comment - It will help to use job.getStatus().getFailureInfo() if the copy job failed? We already get each mapper stack trace in the "main" log if something fail, but I guess is not bad if we report it again at the end. let me add that Another area is to somehow intelligently to estimate the number of copy mappers needed based on the size and number of files? Similar to DistCp? sure, I was thinking to rewrite the file grouping later to keep in consideration also the locations.. but I guess adding a default number of mappers here is short enough
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12631400/HBASE-10622-v0.patch
          against trunk revision .
          ATTACHMENT ID: 12631400

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.

          +1 hadoop1.1. The patch compiles against the hadoop 1.1 profile.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 lineLengths. The patch does not introduce lines longer than 100

          +1 site. The mvn site goal succeeds with this patch.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hbase.client.TestHCM

          -1 core zombie tests. There are 1 zombie test(s):

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/8829//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8829//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8829//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8829//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8829//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8829//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8829//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8829//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8829//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8829//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/8829//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12631400/HBASE-10622-v0.patch against trunk revision . ATTACHMENT ID: 12631400 +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 hadoop1.0 . The patch compiles against the hadoop 1.0 profile. +1 hadoop1.1 . The patch compiles against the hadoop 1.1 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. -1 findbugs . The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 +1 site . The mvn site goal succeeds with this patch. -1 core tests . The patch failed these unit tests: org.apache.hadoop.hbase.client.TestHCM -1 core zombie tests . There are 1 zombie test(s): Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/8829//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8829//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8829//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8829//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8829//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8829//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8829//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8829//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8829//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8829//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/8829//console This message is automatically generated.
          Hide
          Jerry He added a comment -

          It will help to use job.getStatus().getFailureInfo() if the copy job failed?

          Another area is to somehow intelligently to estimate the number of copy mappers needed based on the size and number of files? Similar to DistCp?

          Show
          Jerry He added a comment - It will help to use job.getStatus().getFailureInfo() if the copy job failed? Another area is to somehow intelligently to estimate the number of copy mappers needed based on the size and number of files? Similar to DistCp?
          Hide
          Jonathan Hsieh added a comment -

          This is right on the border of needed to be on review board for me.

          This is a bit awkwards. Make it a do .. while loop?

          +      while (true) {
          +        try {
          +          copyFile(context, inputPath, outputPath);
          +          break;
          +        } catch (IOException e) {
          +          if (++nattempts >= copyRetry) {
          +            throw e;
          +          }
          +          LOG.warn("Failed copy attempt " + nattempts + ". retrying...", e);
          +          Thread.sleep(nattempts * copyRetryWait);
          +        }
          +      }
          

          nit: The user/group may not exist on the

          +        } catch (IOException e) {
          +          LOG.warn("Unable to set the owner/group for file="+ stat.getPath() +": "+ e.getMessage());
          +          LOG.warn("The user/group may not exists on the destination cluster: user=" +
          +                   user + " group=" + group);
          +          return false;
          
          Show
          Jonathan Hsieh added a comment - This is right on the border of needed to be on review board for me. This is a bit awkwards. Make it a do .. while loop? + while ( true ) { + try { + copyFile(context, inputPath, outputPath); + break ; + } catch (IOException e) { + if (++nattempts >= copyRetry) { + throw e; + } + LOG.warn( "Failed copy attempt " + nattempts + ". retrying..." , e); + Thread .sleep(nattempts * copyRetryWait); + } + } nit: The user/group may not exist on the + } catch (IOException e) { + LOG.warn( "Unable to set the owner/group for file=" + stat.getPath() + ": " + e.getMessage()); + LOG.warn( "The user/group may not exists on the destination cluster: user=" + + user + " group=" + group); + return false ;

            People

            • Assignee:
              Matteo Bertozzi
              Reporter:
              Matteo Bertozzi
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development