Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-12239

StorageException complaining " no lease ID" when updating FolderLastModifiedTime in WASB

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.7.0
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: fs/azure, tools
    • Labels:
      None
    • Environment:

      Windows Azure

    • Hadoop Flags:
      Reviewed

      Description

      This is a similar issue as HADOOP-11523 and HADOOP-12089, which I found in a customer's HBase cluster logs, but the piece of code is in a different place.

      2015-07-09 13:38:57,388 INFO org.apache.hadoop.hbase.master.SplitLogManager: dead splitlog workers [workernode3.xxx.b6.internal.cloudapp.net,60020,1436448555180]
      2015-07-09 13:38:57,466 ERROR org.apache.hadoop.hbase.executor.EventHandler: Caught throwable while processing event M_SERVER_SHUTDOWN
      java.io.IOException: failed log splitting for workernode12.xxx.b6.internal.cloudapp.net,60020,1436 448566374, will retry
      	at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.resubmit(ServerShutdownHandler.java:343)
      	at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.process(ServerShutdownHandler.java:211)
      	at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      	at java.lang.Thread.run(Thread.java:745)
      Caused by: java.io.IOException: Unable to write RenamePending file for folder rename from hbase/WALs/workernode12.xxx.b6.internal.cloudapp.net,60020,1436448566374 to hbase/WALs/workernode12.xxx.b6.internal.cloudapp.net,60020,1436448566374-splitting
      	at org.apache.hadoop.fs.azure.NativeAzureFileSystem$FolderRenamePending.writeFile(NativeAzureFileSystem.java:258)
      	at org.apache.hadoop.fs.azure.NativeAzureFileSystem.prepareAtomicFolderRename(NativeAzureFileSystem.java:2110)
      	at org.apache.hadoop.fs.azure.NativeAzureFileSystem.rename(NativeAzureFileSystem.java:1998)
      	at org.apache.hadoop.hbase.master.MasterFileSystem.getLogDirs(MasterFileSystem.java:325)
      	at org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:412)
      	at org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:390)
      	at org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:288)
      	at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.process(ServerShutdownHandler.java:204)
      	... 4 more
      Caused by: org.apache.hadoop.fs.azure.AzureException: com.microsoft.azure.storage.StorageException: There is currently a lease on the blob and no lease ID was specified in the request.
      	at org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.updateFolderLastModifiedTime(AzureNativeFileSystemStore.java:2598)
      	at org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.updateFolderLastModifiedTime(AzureNativeFileSystemStore.java:2609)
      	at org.apache.hadoop.fs.azure.NativeAzureFileSystem.create(NativeAzureFileSystem.java:1366)
      	at org.apache.hadoop.fs.azure.NativeAzureFileSystem.create(NativeAzureFileSystem.java:1195)
      	at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:908)
      	at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:889)
      	at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:786)
      	at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:775)
      	at org.apache.hadoop.fs.azure.NativeAzureFileSystem$FolderRenamePending.writeFile(NativeAzureFileSystem.java:255)
      	... 11 more
      Caused by: com.microsoft.azure.storage.StorageException: There is currently a lease on the blob and no lease ID was specified in the request.
      	at com.microsoft.azure.storage.StorageException.translateException(StorageException.java:89)
      	at com.microsoft.azure.storage.core.StorageRequest.materializeException(StorageRequest.java:307)
      	at com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:182)
      	at com.microsoft.azure.storage.blob.CloudBlob.uploadProperties(CloudBlob.java:2892)
      	at org.apache.hadoop.fs.azure.StorageInterfaceImpl$CloudBlobWrapperImpl.uploadProperties(StorageInterfaceImpl.java:372)
      	at org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.updateFolderLastModifiedTime(AzureNativeFileSystemStore.java:2593)
      	... 19 more
      

        Activity

        Hide
        onpduo Duo Xu added a comment -

        Chris Nauroth

        I think this is the minimum change without touching too much code. Please take a look.

        Show
        onpduo Duo Xu added a comment - Chris Nauroth I think this is the minimum change without touching too much code. Please take a look.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 17m 43s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 javac 8m 49s There were no new javac warning messages.
        +1 javadoc 10m 59s There were no new javadoc warning messages.
        +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
        +1 checkstyle 0m 27s There were no new checkstyle issues.
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 install 1m 28s mvn install still works.
        +1 eclipse:eclipse 0m 36s The patch built with eclipse:eclipse.
        +1 findbugs 0m 52s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        +1 tools/hadoop tests 1m 15s Tests passed in hadoop-azure.
            42m 34s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12745516/HADOOP-12239.01.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / edcaae4
        hadoop-azure test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7286/artifact/patchprocess/testrun_hadoop-azure.txt
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7286/testReport/
        Java 1.7.0_55
        uname Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7286/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 17m 43s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 8m 49s There were no new javac warning messages. +1 javadoc 10m 59s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 27s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 28s mvn install still works. +1 eclipse:eclipse 0m 36s The patch built with eclipse:eclipse. +1 findbugs 0m 52s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 tools/hadoop tests 1m 15s Tests passed in hadoop-azure.     42m 34s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12745516/HADOOP-12239.01.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / edcaae4 hadoop-azure test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7286/artifact/patchprocess/testrun_hadoop-azure.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7286/testReport/ Java 1.7.0_55 uname Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7286/console This message was automatically generated.
        Hide
        cnauroth Chris Nauroth added a comment -

        Hello Duo Xu. The patch looks good to me. I also looked around for other bare calls to updateFolderLastModifiedTime that aren't guarded by the lease, and I didn't find any. Hopefully that means these issues are completely fixed now.

        The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.

        I understand that it would be extremely complex to try to cover this in a unit test. Can you please describe any manual testing that you've done? Thanks!

        Show
        cnauroth Chris Nauroth added a comment - Hello Duo Xu . The patch looks good to me. I also looked around for other bare calls to updateFolderLastModifiedTime that aren't guarded by the lease, and I didn't find any. Hopefully that means these issues are completely fixed now. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. I understand that it would be extremely complex to try to cover this in a unit test. Can you please describe any manual testing that you've done? Thanks!
        Hide
        onpduo Duo Xu added a comment -

        Chris Nauroth

        I compiled the code and replaced the jar on an existing cluster, then I ran YCSB to ingest data for hours and rebooted some nodes, distributed log splitting worked fine.

        Show
        onpduo Duo Xu added a comment - Chris Nauroth I compiled the code and replaced the jar on an existing cluster, then I ran YCSB to ingest data for hours and rebooted some nodes, distributed log splitting worked fine.
        Hide
        cnauroth Chris Nauroth added a comment -

        +1 for the patch. I have committed this to trunk and branch-2. Duo, thank you for another contribution!

        Show
        cnauroth Chris Nauroth added a comment - +1 for the patch. I have committed this to trunk and branch-2. Duo, thank you for another contribution!
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #8201 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8201/)
        HADOOP-12239. StorageException complaining " no lease ID" when updating FolderLastModifiedTime in WASB. Contributed by Duo Xu. (cnauroth: rev efa97243ecb84b3b468e732897cd685e3869f480)

        • hadoop-common-project/hadoop-common/CHANGES.txt
        • hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/NativeAzureFileSystem.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8201 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8201/ ) HADOOP-12239 . StorageException complaining " no lease ID" when updating FolderLastModifiedTime in WASB. Contributed by Duo Xu. (cnauroth: rev efa97243ecb84b3b468e732897cd685e3869f480) hadoop-common-project/hadoop-common/CHANGES.txt hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/NativeAzureFileSystem.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk #995 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/995/)
        HADOOP-12239. StorageException complaining " no lease ID" when updating FolderLastModifiedTime in WASB. Contributed by Duo Xu. (cnauroth: rev efa97243ecb84b3b468e732897cd685e3869f480)

        • hadoop-common-project/hadoop-common/CHANGES.txt
        • hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/NativeAzureFileSystem.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #995 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/995/ ) HADOOP-12239 . StorageException complaining " no lease ID" when updating FolderLastModifiedTime in WASB. Contributed by Duo Xu. (cnauroth: rev efa97243ecb84b3b468e732897cd685e3869f480) hadoop-common-project/hadoop-common/CHANGES.txt hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/NativeAzureFileSystem.java
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #265 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/265/)
        HADOOP-12239. StorageException complaining " no lease ID" when updating FolderLastModifiedTime in WASB. Contributed by Duo Xu. (cnauroth: rev efa97243ecb84b3b468e732897cd685e3869f480)

        • hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/NativeAzureFileSystem.java
        • hadoop-common-project/hadoop-common/CHANGES.txt
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #265 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/265/ ) HADOOP-12239 . StorageException complaining " no lease ID" when updating FolderLastModifiedTime in WASB. Contributed by Duo Xu. (cnauroth: rev efa97243ecb84b3b468e732897cd685e3869f480) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/NativeAzureFileSystem.java hadoop-common-project/hadoop-common/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk #2192 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2192/)
        HADOOP-12239. StorageException complaining " no lease ID" when updating FolderLastModifiedTime in WASB. Contributed by Duo Xu. (cnauroth: rev efa97243ecb84b3b468e732897cd685e3869f480)

        • hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/NativeAzureFileSystem.java
        • hadoop-common-project/hadoop-common/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2192 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2192/ ) HADOOP-12239 . StorageException complaining " no lease ID" when updating FolderLastModifiedTime in WASB. Contributed by Duo Xu. (cnauroth: rev efa97243ecb84b3b468e732897cd685e3869f480) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/NativeAzureFileSystem.java hadoop-common-project/hadoop-common/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #254 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/254/)
        HADOOP-12239. StorageException complaining " no lease ID" when updating FolderLastModifiedTime in WASB. Contributed by Duo Xu. (cnauroth: rev efa97243ecb84b3b468e732897cd685e3869f480)

        • hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/NativeAzureFileSystem.java
        • hadoop-common-project/hadoop-common/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #254 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/254/ ) HADOOP-12239 . StorageException complaining " no lease ID" when updating FolderLastModifiedTime in WASB. Contributed by Duo Xu. (cnauroth: rev efa97243ecb84b3b468e732897cd685e3869f480) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/NativeAzureFileSystem.java hadoop-common-project/hadoop-common/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #262 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/262/)
        HADOOP-12239. StorageException complaining " no lease ID" when updating FolderLastModifiedTime in WASB. Contributed by Duo Xu. (cnauroth: rev efa97243ecb84b3b468e732897cd685e3869f480)

        • hadoop-common-project/hadoop-common/CHANGES.txt
        • hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/NativeAzureFileSystem.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #262 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/262/ ) HADOOP-12239 . StorageException complaining " no lease ID" when updating FolderLastModifiedTime in WASB. Contributed by Duo Xu. (cnauroth: rev efa97243ecb84b3b468e732897cd685e3869f480) hadoop-common-project/hadoop-common/CHANGES.txt hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/NativeAzureFileSystem.java
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2211 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2211/)
        HADOOP-12239. StorageException complaining " no lease ID" when updating FolderLastModifiedTime in WASB. Contributed by Duo Xu. (cnauroth: rev efa97243ecb84b3b468e732897cd685e3869f480)

        • hadoop-common-project/hadoop-common/CHANGES.txt
        • hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/NativeAzureFileSystem.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2211 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2211/ ) HADOOP-12239 . StorageException complaining " no lease ID" when updating FolderLastModifiedTime in WASB. Contributed by Duo Xu. (cnauroth: rev efa97243ecb84b3b468e732897cd685e3869f480) hadoop-common-project/hadoop-common/CHANGES.txt hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/NativeAzureFileSystem.java

          People

          • Assignee:
            onpduo Duo Xu
            Reporter:
            onpduo Duo Xu
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development