Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-12239

StorageException complaining " no lease ID" when updating FolderLastModifiedTime in WASB

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.7.0
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: fs/azure, tools
    • Labels:
      None
    • Environment:

      Windows Azure

    • Hadoop Flags:
      Reviewed

      Description

      This is a similar issue as HADOOP-11523 and HADOOP-12089, which I found in a customer's HBase cluster logs, but the piece of code is in a different place.

      2015-07-09 13:38:57,388 INFO org.apache.hadoop.hbase.master.SplitLogManager: dead splitlog workers [workernode3.xxx.b6.internal.cloudapp.net,60020,1436448555180]
      2015-07-09 13:38:57,466 ERROR org.apache.hadoop.hbase.executor.EventHandler: Caught throwable while processing event M_SERVER_SHUTDOWN
      java.io.IOException: failed log splitting for workernode12.xxx.b6.internal.cloudapp.net,60020,1436 448566374, will retry
      	at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.resubmit(ServerShutdownHandler.java:343)
      	at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.process(ServerShutdownHandler.java:211)
      	at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      	at java.lang.Thread.run(Thread.java:745)
      Caused by: java.io.IOException: Unable to write RenamePending file for folder rename from hbase/WALs/workernode12.xxx.b6.internal.cloudapp.net,60020,1436448566374 to hbase/WALs/workernode12.xxx.b6.internal.cloudapp.net,60020,1436448566374-splitting
      	at org.apache.hadoop.fs.azure.NativeAzureFileSystem$FolderRenamePending.writeFile(NativeAzureFileSystem.java:258)
      	at org.apache.hadoop.fs.azure.NativeAzureFileSystem.prepareAtomicFolderRename(NativeAzureFileSystem.java:2110)
      	at org.apache.hadoop.fs.azure.NativeAzureFileSystem.rename(NativeAzureFileSystem.java:1998)
      	at org.apache.hadoop.hbase.master.MasterFileSystem.getLogDirs(MasterFileSystem.java:325)
      	at org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:412)
      	at org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:390)
      	at org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:288)
      	at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.process(ServerShutdownHandler.java:204)
      	... 4 more
      Caused by: org.apache.hadoop.fs.azure.AzureException: com.microsoft.azure.storage.StorageException: There is currently a lease on the blob and no lease ID was specified in the request.
      	at org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.updateFolderLastModifiedTime(AzureNativeFileSystemStore.java:2598)
      	at org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.updateFolderLastModifiedTime(AzureNativeFileSystemStore.java:2609)
      	at org.apache.hadoop.fs.azure.NativeAzureFileSystem.create(NativeAzureFileSystem.java:1366)
      	at org.apache.hadoop.fs.azure.NativeAzureFileSystem.create(NativeAzureFileSystem.java:1195)
      	at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:908)
      	at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:889)
      	at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:786)
      	at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:775)
      	at org.apache.hadoop.fs.azure.NativeAzureFileSystem$FolderRenamePending.writeFile(NativeAzureFileSystem.java:255)
      	... 11 more
      Caused by: com.microsoft.azure.storage.StorageException: There is currently a lease on the blob and no lease ID was specified in the request.
      	at com.microsoft.azure.storage.StorageException.translateException(StorageException.java:89)
      	at com.microsoft.azure.storage.core.StorageRequest.materializeException(StorageRequest.java:307)
      	at com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:182)
      	at com.microsoft.azure.storage.blob.CloudBlob.uploadProperties(CloudBlob.java:2892)
      	at org.apache.hadoop.fs.azure.StorageInterfaceImpl$CloudBlobWrapperImpl.uploadProperties(StorageInterfaceImpl.java:372)
      	at org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.updateFolderLastModifiedTime(AzureNativeFileSystemStore.java:2593)
      	... 19 more
      

        Attachments

          Activity

            People

            • Assignee:
              onpduo Duo Xu
              Reporter:
              onpduo Duo Xu
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: