HBase
  1. HBase
  2. HBASE-11119

Update ExportSnapShot to optionally not use a tmp file on external file system

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.99.0, 0.96.3, 0.94.20, 0.98.3
    • Component/s: snapshots
    • Labels:
      None

      Description

      There are FileSystem like S3 where renaming is extremely expensive. This patch will add a parameter that says something like

      use.tmp.folder

      It will be defaulted to true. So default behavior is the same. If false is set them the files will land in the final destination with no need for a rename.

        Activity

        Ted Malaska created issue -
        Matteo Bertozzi made changes -
        Field Original Value New Value
        Assignee Ted Malaska [ malaskat ]
        Hide
        Ted Malaska added a comment -

        Added the support for skip the rename of the tmp directory.

        This is supported by a new config named "snapshot.export.skip.tmp" which is default to false. So the existing behavior is the default.

        I tried to touch as little lines as possible.

        I also added a new unit test.

        The code compiles but I'm unable to execute the test on my box. I will try to resolve that tomorrow. But the issue seems to be unrelated to my changes.

        Show
        Ted Malaska added a comment - Added the support for skip the rename of the tmp directory. This is supported by a new config named "snapshot.export.skip.tmp" which is default to false. So the existing behavior is the default. I tried to touch as little lines as possible. I also added a new unit test. The code compiles but I'm unable to execute the test on my box. I will try to resolve that tomorrow. But the issue seems to be unrelated to my changes.
        Ted Malaska made changes -
        Attachment HBASE-11119.patch [ 12643889 ]
        Ted Malaska made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12643889/HBASE-11119.patch
        against trunk revision .
        ATTACHMENT ID: 12643889

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 lineLengths. The patch introduces the following lines longer than 100:
        + System.err.println("Unable to remove existing snapshot tmp directory: " + snapshotTmpDir);
        + System.err.println("A snapshot with the same name '"+ snapshotName +"' may be in-progress");
        + System.err.println("Please check " + snapshotTmpDir + ". If the snapshot has completed, ");
        + System.err.println("consider removing "+ snapshotTmpDir +" by using the -overwrite option");

        +1 site. The mvn site goal succeeds with this patch.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.hbase.client.TestMultiParallel

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/9481//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9481//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9481//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9481//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9481//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9481//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9481//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9481//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9481//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9481//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/9481//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12643889/HBASE-11119.patch against trunk revision . ATTACHMENT ID: 12643889 +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 3 new or modified tests. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 lineLengths . The patch introduces the following lines longer than 100: + System.err.println("Unable to remove existing snapshot tmp directory: " + snapshotTmpDir); + System.err.println("A snapshot with the same name '"+ snapshotName +"' may be in-progress"); + System.err.println("Please check " + snapshotTmpDir + ". If the snapshot has completed, "); + System.err.println("consider removing "+ snapshotTmpDir +" by using the -overwrite option"); +1 site . The mvn site goal succeeds with this patch. -1 core tests . The patch failed these unit tests: org.apache.hadoop.hbase.client.TestMultiParallel Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/9481//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9481//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9481//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9481//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9481//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9481//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9481//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9481//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9481//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9481//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/9481//console This message is automatically generated.
        Matteo Bertozzi made changes -
        Issue Type New Feature [ 2 ] Improvement [ 4 ]
        Fix Version/s 0.99.0 [ 12325675 ]
        Fix Version/s 0.96.3 [ 12326538 ]
        Fix Version/s 0.94.20 [ 12326654 ]
        Fix Version/s 0.98.3 [ 12326765 ]
        Component/s snapshots [ 12319654 ]
        Hide
        Matteo Bertozzi added a comment -

        +1 patch looks good to me, it applies cleanly to 96, 98 and trunk and almost cleanly to 94.
        I'll fix the long lines problem on commit.

        (ok to have a -Dsnapshot.export.skip.tmp=true instead of a -skip-tmp because you must know what you are doing. This may speedup a bit where the rename operation is slow but you have to manually handle the in-progress or failure case, since for the tools the snapshot will be available as soon as the manifest is copied)

        Show
        Matteo Bertozzi added a comment - +1 patch looks good to me, it applies cleanly to 96, 98 and trunk and almost cleanly to 94. I'll fix the long lines problem on commit. (ok to have a -Dsnapshot.export.skip.tmp=true instead of a -skip-tmp because you must know what you are doing. This may speedup a bit where the rename operation is slow but you have to manually handle the in-progress or failure case, since for the tools the snapshot will be available as soon as the manifest is copied)
        Matteo Bertozzi made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in HBase-0.98 #303 (See https://builds.apache.org/job/HBase-0.98/303/)
        HBASE-11119 Update ExportSnapShot to optionally not use a tmp file on external file system (Ted Malaska) (mbertozzi: rev 1593338)

        • /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshot.java
        • /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshot.java
        Show
        Hudson added a comment - SUCCESS: Integrated in HBase-0.98 #303 (See https://builds.apache.org/job/HBase-0.98/303/ ) HBASE-11119 Update ExportSnapShot to optionally not use a tmp file on external file system (Ted Malaska) (mbertozzi: rev 1593338) /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshot.java /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshot.java
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in HBase-0.94-security #481 (See https://builds.apache.org/job/HBase-0.94-security/481/)
        HBASE-11119 Update ExportSnapShot to optionally not use a tmp file on external file system (Ted Malaska) (mbertozzi: rev 1593340)

        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshot.java
        • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshot.java
        Show
        Hudson added a comment - SUCCESS: Integrated in HBase-0.94-security #481 (See https://builds.apache.org/job/HBase-0.94-security/481/ ) HBASE-11119 Update ExportSnapShot to optionally not use a tmp file on external file system (Ted Malaska) (mbertozzi: rev 1593340) /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshot.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshot.java
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in HBase-0.94 #1366 (See https://builds.apache.org/job/HBase-0.94/1366/)
        HBASE-11119 Update ExportSnapShot to optionally not use a tmp file on external file system (Ted Malaska) (mbertozzi: rev 1593340)

        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshot.java
        • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshot.java
        Show
        Hudson added a comment - SUCCESS: Integrated in HBase-0.94 #1366 (See https://builds.apache.org/job/HBase-0.94/1366/ ) HBASE-11119 Update ExportSnapShot to optionally not use a tmp file on external file system (Ted Malaska) (mbertozzi: rev 1593340) /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshot.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshot.java
        Hide
        Hudson added a comment -

        FAILURE: Integrated in HBase-0.94-on-Hadoop-2 #83 (See https://builds.apache.org/job/HBase-0.94-on-Hadoop-2/83/)
        HBASE-11119 Update ExportSnapShot to optionally not use a tmp file on external file system (Ted Malaska) (mbertozzi: rev 1593340)

        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshot.java
        • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshot.java
        Show
        Hudson added a comment - FAILURE: Integrated in HBase-0.94-on-Hadoop-2 #83 (See https://builds.apache.org/job/HBase-0.94-on-Hadoop-2/83/ ) HBASE-11119 Update ExportSnapShot to optionally not use a tmp file on external file system (Ted Malaska) (mbertozzi: rev 1593340) /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshot.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshot.java
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in HBase-0.94-JDK7 #133 (See https://builds.apache.org/job/HBase-0.94-JDK7/133/)
        HBASE-11119 Update ExportSnapShot to optionally not use a tmp file on external file system (Ted Malaska) (mbertozzi: rev 1593340)

        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshot.java
        • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshot.java
        Show
        Hudson added a comment - SUCCESS: Integrated in HBase-0.94-JDK7 #133 (See https://builds.apache.org/job/HBase-0.94-JDK7/133/ ) HBASE-11119 Update ExportSnapShot to optionally not use a tmp file on external file system (Ted Malaska) (mbertozzi: rev 1593340) /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshot.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshot.java
        Hide
        Hudson added a comment -

        FAILURE: Integrated in HBase-TRUNK #5131 (See https://builds.apache.org/job/HBase-TRUNK/5131/)
        HBASE-11119 Update ExportSnapShot to optionally not use a tmp file on external file system (Ted Malaska) (mbertozzi: rev 1593337)

        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshot.java
        • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshot.java
        Show
        Hudson added a comment - FAILURE: Integrated in HBase-TRUNK #5131 (See https://builds.apache.org/job/HBase-TRUNK/5131/ ) HBASE-11119 Update ExportSnapShot to optionally not use a tmp file on external file system (Ted Malaska) (mbertozzi: rev 1593337) /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshot.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshot.java
        Hide
        Hudson added a comment -

        FAILURE: Integrated in HBase-0.98-on-Hadoop-1.1 #288 (See https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/288/)
        HBASE-11119 Update ExportSnapShot to optionally not use a tmp file on external file system (Ted Malaska) (mbertozzi: rev 1593338)

        • /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshot.java
        • /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshot.java
        Show
        Hudson added a comment - FAILURE: Integrated in HBase-0.98-on-Hadoop-1.1 #288 (See https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/288/ ) HBASE-11119 Update ExportSnapShot to optionally not use a tmp file on external file system (Ted Malaska) (mbertozzi: rev 1593338) /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshot.java /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshot.java
        Hide
        Hudson added a comment -

        FAILURE: Integrated in hbase-0.96 #397 (See https://builds.apache.org/job/hbase-0.96/397/)
        HBASE-11119 Update ExportSnapShot to optionally not use a tmp file on external file system (Ted Malaska) (mbertozzi: rev 1593339)

        • /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshot.java
        • /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshot.java
        Show
        Hudson added a comment - FAILURE: Integrated in hbase-0.96 #397 (See https://builds.apache.org/job/hbase-0.96/397/ ) HBASE-11119 Update ExportSnapShot to optionally not use a tmp file on external file system (Ted Malaska) (mbertozzi: rev 1593339) /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshot.java /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshot.java
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in hbase-0.96-hadoop2 #276 (See https://builds.apache.org/job/hbase-0.96-hadoop2/276/)
        HBASE-11119 Update ExportSnapShot to optionally not use a tmp file on external file system (Ted Malaska) (mbertozzi: rev 1593339)

        • /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshot.java
        • /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshot.java
        Show
        Hudson added a comment - SUCCESS: Integrated in hbase-0.96-hadoop2 #276 (See https://builds.apache.org/job/hbase-0.96-hadoop2/276/ ) HBASE-11119 Update ExportSnapShot to optionally not use a tmp file on external file system (Ted Malaska) (mbertozzi: rev 1593339) /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshot.java /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshot.java
        Lars Hofhansl made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            Ted Malaska
            Reporter:
            Ted Malaska
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development