Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-11523

StorageException complaining " no lease ID" when updating FolderLastModifiedTime in WASB

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.7.0
    • Component/s: tools
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      In current WASB (Windows Azure Storage - Blob) implementation, when rename operation succeeds, WASB will update the parent folder's "last modified time" property. By default we do not acquire lease on this folder when updating its property and simply pass "null" to it.

      In HBase scenario, when doing distributed log splitting, there might be a case that multiple processes from different region servers will access the same folder, and randomly we will see this exception in regionserver's log, which makes log splitting fail.

      So we should acquire the lease when updating the folder property rather than pass "null" to it.

      ERROR org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Couldn't rename wasb://xxx/hbase/data/default/tdelrowtbl/3c842e8823c192d1028dc72ac3f22886/recovered.edits/0000000000000000015.temp to wasb://xxx/hbase/data/default/tdelrowtbl/3c842e8823c192d1028dc72ac3f22886/recovered.edits/0000000000000000015
      org.apache.hadoop.fs.azure.AzureException: com.microsoft.windowsazure.storage.StorageException: There is currently a lease on the blob and no lease ID was specified in the request.
      	at org.apache.hadoop.fs.azurenative.AzureNativeFileSystemStore.updateFolderLastModifiedTime(AzureNativeFileSystemStore.java:2558)
      	at org.apache.hadoop.fs.azurenative.AzureNativeFileSystemStore.updateFolderLastModifiedTime(AzureNativeFileSystemStore.java:2569)
      	at org.apache.hadoop.fs.azurenative.NativeAzureFileSystem.updateParentFolderLastModifiedTime(NativeAzureFileSystem.java:2016)
      	at org.apache.hadoop.fs.azurenative.NativeAzureFileSystem.rename(NativeAzureFileSystem.java:1983)
      	at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter$LogRecoveredEditsOutputSink$2.call(HLogSplitter.java:1161)
      	at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter$LogRecoveredEditsOutputSink$2.call(HLogSplitter.java:1121)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      	at java.lang.Thread.run(Thread.java:745)
      

        Attachments

        1. HADOOP-11523.1.patch
          2 kB
          Duo Xu
        2. HADOOP-11523.2.patch
          2 kB
          Duo Xu
        3. HADOOP-11523.3.patch
          2 kB
          Duo Xu
        4. HADOOP-11523.4.patch
          2 kB
          Duo Xu

          Issue Links

            Activity

              People

              • Assignee:
                onpduo Duo Xu
                Reporter:
                onpduo Duo Xu
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: