Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-12780

During WASB atomic rename handle crash when one directory has been renamed but not file under it.

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 2.8.0
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: fs/azure
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      During atomic folder rename process preperaion we record the proposed change to a metadata file (-renamePending.json).
      Say we are renaming parent/folderToRename to parent/renamedFolder.
      folderToRename has an inner folder innerFolder and innerFolder has a file innerFile
      Content of the –renamePending.json file will be

      { OldFolderName: parent/ folderToRename", NewFolderName: "parent/renamedFolder", FileList: [ "innerFolder", "innerFolder/innerFile" ] }

      Atfirst we rename all files within the source directory and then rename the source directory at the last step
      The steps are
      1. Atfirst we will rename innerFolder,
      2. Then rename innerFolder/innerFile
      3. Then rename source directory folderToRename
      Say the process crashes after step 1.
      So innerFolder has been renamed.
      Note that Azure storage does not natively support folder. So if a directory created by mkdir command, we create an empty placeholder blob with metadata for the directory.
      So after step 1, the empty blob corresponding to the directory innerFolder has been renamed.
      When the process comes up, in redo path it will go through the –renamePending.json file try to redo the renames.
      For each file in file list of renamePending file it checks if the source file exists, if source file exists then it renames the file. When it gets innerFolder, it calls filesystem.exists(innerFolder). Now filesystem.exists(innerFolder) will return true, because file under that folder exists even though the empty blob corresponding th that folder does not exist. So it will try to rename this folder, and as the empty blob has already been deleted so this fails with exception that “source blob does not exist”.

      1. HADOOP-12780.001.patch
        8 kB
        madhumita chakraborty

        Issue Links

          Activity

          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 11s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 7m 21s trunk passed
          +1 compile 0m 14s trunk passed with JDK v1.8.0_66
          +1 compile 0m 16s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 14s trunk passed
          +1 mvnsite 0m 22s trunk passed
          +1 mvneclipse 0m 12s trunk passed
          +1 findbugs 0m 32s trunk passed
          +1 javadoc 0m 13s trunk passed with JDK v1.8.0_66
          +1 javadoc 0m 14s trunk passed with JDK v1.7.0_91
          +1 mvninstall 0m 16s the patch passed
          +1 compile 0m 12s the patch passed with JDK v1.8.0_66
          +1 javac 0m 12s the patch passed
          +1 compile 0m 14s the patch passed with JDK v1.7.0_91
          +1 javac 0m 14s the patch passed
          -1 checkstyle 0m 12s hadoop-tools/hadoop-azure: patch generated 1 new + 45 unchanged - 0 fixed = 46 total (was 45)
          +1 mvnsite 0m 20s the patch passed
          +1 mvneclipse 0m 9s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 0m 41s the patch passed
          +1 javadoc 0m 10s the patch passed with JDK v1.8.0_66
          +1 javadoc 0m 12s the patch passed with JDK v1.7.0_91
          +1 unit 1m 7s hadoop-azure in the patch passed with JDK v1.8.0_66.
          +1 unit 1m 22s hadoop-azure in the patch passed with JDK v1.7.0_91.
          +1 asflicense 0m 19s Patch does not generate ASF License warnings.
          16m 4s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12786990/HADOOP-12780.001.patch
          JIRA Issue HADOOP-12780
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux fe906c2f6b36 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / acac729
          Default Java 1.7.0_91
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/8574/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-azure.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8574/testReport/
          modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
          Max memory used 77MB
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8574/console
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 11s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 21s trunk passed +1 compile 0m 14s trunk passed with JDK v1.8.0_66 +1 compile 0m 16s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 14s trunk passed +1 mvnsite 0m 22s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 0m 32s trunk passed +1 javadoc 0m 13s trunk passed with JDK v1.8.0_66 +1 javadoc 0m 14s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 16s the patch passed +1 compile 0m 12s the patch passed with JDK v1.8.0_66 +1 javac 0m 12s the patch passed +1 compile 0m 14s the patch passed with JDK v1.7.0_91 +1 javac 0m 14s the patch passed -1 checkstyle 0m 12s hadoop-tools/hadoop-azure: patch generated 1 new + 45 unchanged - 0 fixed = 46 total (was 45) +1 mvnsite 0m 20s the patch passed +1 mvneclipse 0m 9s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 0m 41s the patch passed +1 javadoc 0m 10s the patch passed with JDK v1.8.0_66 +1 javadoc 0m 12s the patch passed with JDK v1.7.0_91 +1 unit 1m 7s hadoop-azure in the patch passed with JDK v1.8.0_66. +1 unit 1m 22s hadoop-azure in the patch passed with JDK v1.7.0_91. +1 asflicense 0m 19s Patch does not generate ASF License warnings. 16m 4s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12786990/HADOOP-12780.001.patch JIRA Issue HADOOP-12780 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux fe906c2f6b36 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / acac729 Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/8574/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-azure.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8574/testReport/ modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure Max memory used 77MB Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8574/console Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 10s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 7m 5s trunk passed
          +1 compile 0m 14s trunk passed with JDK v1.8.0_66
          +1 compile 0m 16s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 14s trunk passed
          +1 mvnsite 0m 22s trunk passed
          +1 mvneclipse 0m 12s trunk passed
          +1 findbugs 0m 30s trunk passed
          +1 javadoc 0m 13s trunk passed with JDK v1.8.0_66
          +1 javadoc 0m 14s trunk passed with JDK v1.7.0_91
          +1 mvninstall 0m 16s the patch passed
          +1 compile 0m 12s the patch passed with JDK v1.8.0_66
          +1 javac 0m 12s the patch passed
          +1 compile 0m 14s the patch passed with JDK v1.7.0_91
          +1 javac 0m 14s the patch passed
          +1 checkstyle 0m 12s the patch passed
          +1 mvnsite 0m 19s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 0m 42s the patch passed
          +1 javadoc 0m 10s the patch passed with JDK v1.8.0_66
          +1 javadoc 0m 12s the patch passed with JDK v1.7.0_91
          +1 unit 1m 7s hadoop-azure in the patch passed with JDK v1.8.0_66.
          +1 unit 1m 22s hadoop-azure in the patch passed with JDK v1.7.0_91.
          +1 asflicense 0m 17s Patch does not generate ASF License warnings.
          15m 45s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12786995/HADOOP-12780.001.patch
          JIRA Issue HADOOP-12780
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 065bbaccf7e7 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / acac729
          Default Java 1.7.0_91
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
          findbugs v3.0.0
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8575/testReport/
          modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
          Max memory used 76MB
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8575/console
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 10s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 5s trunk passed +1 compile 0m 14s trunk passed with JDK v1.8.0_66 +1 compile 0m 16s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 14s trunk passed +1 mvnsite 0m 22s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 0m 30s trunk passed +1 javadoc 0m 13s trunk passed with JDK v1.8.0_66 +1 javadoc 0m 14s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 16s the patch passed +1 compile 0m 12s the patch passed with JDK v1.8.0_66 +1 javac 0m 12s the patch passed +1 compile 0m 14s the patch passed with JDK v1.7.0_91 +1 javac 0m 14s the patch passed +1 checkstyle 0m 12s the patch passed +1 mvnsite 0m 19s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 0m 42s the patch passed +1 javadoc 0m 10s the patch passed with JDK v1.8.0_66 +1 javadoc 0m 12s the patch passed with JDK v1.7.0_91 +1 unit 1m 7s hadoop-azure in the patch passed with JDK v1.8.0_66. +1 unit 1m 22s hadoop-azure in the patch passed with JDK v1.7.0_91. +1 asflicense 0m 17s Patch does not generate ASF License warnings. 15m 45s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12786995/HADOOP-12780.001.patch JIRA Issue HADOOP-12780 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 065bbaccf7e7 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / acac729 Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8575/testReport/ modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure Max memory used 76MB Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8575/console Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          madhuch-ms madhumita chakraborty added a comment -

          Chris Nauroth Could you please take a look at the patch?

          Show
          madhuch-ms madhumita chakraborty added a comment - Chris Nauroth Could you please take a look at the patch?
          Hide
          cnauroth Chris Nauroth added a comment -

          madhumita chakraborty, thank you for the patch. This is a nice find.

          +1. I have committed this to trunk, branch-2 and branch-2.8.

          Show
          cnauroth Chris Nauroth added a comment - madhumita chakraborty , thank you for the patch. This is a nice find. +1. I have committed this to trunk, branch-2 and branch-2.8.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #9299 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9299/)
          HADOOP-12780. During WASB atomic rename handle crash when one directory (cnauroth: rev 91a96eaa534dbb27e81b6c24bbb8138200a80a83)

          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/AzureNativeFileSystemStore.java
          • hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/NativeFileSystemStore.java
          • hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/NativeAzureFileSystemBaseTest.java
          • hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/NativeAzureFileSystem.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9299 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9299/ ) HADOOP-12780 . During WASB atomic rename handle crash when one directory (cnauroth: rev 91a96eaa534dbb27e81b6c24bbb8138200a80a83) hadoop-common-project/hadoop-common/CHANGES.txt hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/AzureNativeFileSystemStore.java hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/NativeFileSystemStore.java hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/NativeAzureFileSystemBaseTest.java hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/NativeAzureFileSystem.java

            People

            • Assignee:
              madhuch-ms madhumita chakraborty
              Reporter:
              madhuch-ms madhumita chakraborty
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development