Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-12508

delete fails with exception when lease is held on blob

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: fs/azure
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The delete function as implemented by AzureNativeFileSystem store attempts delete without a lease. In most cases this works but in the case of a dangling lease resulting out of say a process killed and leaving a lease dangling for a small period a delete attempted during this period simply crashes. This fix addresses the situation by re-attempting the delete after a lease acqusition in this case

      1. HADOOP-12508.06.patch
        7 kB
        Chris Nauroth
      2. HADOOP-12508.05.patch
        5 kB
        Gaurav Kanade
      3. HADOOP-12508.04.patch
        5 kB
        Gaurav Kanade
      4. HADOOP-12508.03.patch
        4 kB
        Gaurav Kanade
      5. HADOOP-12508.02.patch
        2 kB
        Gaurav Kanade
      6. HADOOP-12508.01.patch
        2 kB
        Gaurav Kanade

        Issue Links

          Activity

          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2501 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2501/)
          HADOOP-12508. delete fails with exception when lease is held on blob. (cnauroth: rev 9e7dcab185abf2fdabb28f2799b9952b5664a4b0)

          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/SelfRenewingLease.java
          • hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestNativeAzureFileSystemLive.java
          • hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/AzureNativeFileSystemStore.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2501 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2501/ ) HADOOP-12508 . delete fails with exception when lease is held on blob. (cnauroth: rev 9e7dcab185abf2fdabb28f2799b9952b5664a4b0) hadoop-common-project/hadoop-common/CHANGES.txt hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/SelfRenewingLease.java hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestNativeAzureFileSystemLive.java hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/AzureNativeFileSystemStore.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #564 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/564/)
          HADOOP-12508. delete fails with exception when lease is held on blob. (cnauroth: rev 9e7dcab185abf2fdabb28f2799b9952b5664a4b0)

          • hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestNativeAzureFileSystemLive.java
          • hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/SelfRenewingLease.java
          • hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/AzureNativeFileSystemStore.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #564 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/564/ ) HADOOP-12508 . delete fails with exception when lease is held on blob. (cnauroth: rev 9e7dcab185abf2fdabb28f2799b9952b5664a4b0) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestNativeAzureFileSystemLive.java hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/SelfRenewingLease.java hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/AzureNativeFileSystemStore.java hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2559 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2559/)
          HADOOP-12508. delete fails with exception when lease is held on blob. (cnauroth: rev 9e7dcab185abf2fdabb28f2799b9952b5664a4b0)

          • hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/SelfRenewingLease.java
          • hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestNativeAzureFileSystemLive.java
          • hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/AzureNativeFileSystemStore.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2559 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2559/ ) HADOOP-12508 . delete fails with exception when lease is held on blob. (cnauroth: rev 9e7dcab185abf2fdabb28f2799b9952b5664a4b0) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/SelfRenewingLease.java hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestNativeAzureFileSystemLive.java hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/AzureNativeFileSystemStore.java hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #1351 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1351/)
          HADOOP-12508. delete fails with exception when lease is held on blob. (cnauroth: rev 9e7dcab185abf2fdabb28f2799b9952b5664a4b0)

          • hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/AzureNativeFileSystemStore.java
          • hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/SelfRenewingLease.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestNativeAzureFileSystemLive.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #1351 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1351/ ) HADOOP-12508 . delete fails with exception when lease is held on blob. (cnauroth: rev 9e7dcab185abf2fdabb28f2799b9952b5664a4b0) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/AzureNativeFileSystemStore.java hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/SelfRenewingLease.java hadoop-common-project/hadoop-common/CHANGES.txt hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestNativeAzureFileSystemLive.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #628 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/628/)
          HADOOP-12508. delete fails with exception when lease is held on blob. (cnauroth: rev 9e7dcab185abf2fdabb28f2799b9952b5664a4b0)

          • hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/SelfRenewingLease.java
          • hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/AzureNativeFileSystemStore.java
          • hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestNativeAzureFileSystemLive.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #628 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/628/ ) HADOOP-12508 . delete fails with exception when lease is held on blob. (cnauroth: rev 9e7dcab185abf2fdabb28f2799b9952b5664a4b0) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/SelfRenewingLease.java hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/AzureNativeFileSystemStore.java hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestNativeAzureFileSystemLive.java hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #617 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/617/)
          HADOOP-12508. delete fails with exception when lease is held on blob. (cnauroth: rev 9e7dcab185abf2fdabb28f2799b9952b5664a4b0)

          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/SelfRenewingLease.java
          • hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestNativeAzureFileSystemLive.java
          • hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/AzureNativeFileSystemStore.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #617 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/617/ ) HADOOP-12508 . delete fails with exception when lease is held on blob. (cnauroth: rev 9e7dcab185abf2fdabb28f2799b9952b5664a4b0) hadoop-common-project/hadoop-common/CHANGES.txt hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/SelfRenewingLease.java hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestNativeAzureFileSystemLive.java hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/AzureNativeFileSystemStore.java
          Hide
          cnauroth Chris Nauroth added a comment -

          This patch introduced an intermittent test failure in one of the live tests that connects to a real Azure Storage account: TestAzureFileSystemInstrumentation#testClientErrorMetrics. I have assigned HADOOP-12540 to Gaurav Kanade for follow-up.

          Show
          cnauroth Chris Nauroth added a comment - This patch introduced an intermittent test failure in one of the live tests that connects to a real Azure Storage account: TestAzureFileSystemInstrumentation#testClientErrorMetrics . I have assigned HADOOP-12540 to Gaurav Kanade for follow-up.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #8743 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8743/)
          HADOOP-12508. delete fails with exception when lease is held on blob. (cnauroth: rev 9e7dcab185abf2fdabb28f2799b9952b5664a4b0)

          • hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/SelfRenewingLease.java
          • hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestNativeAzureFileSystemLive.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/AzureNativeFileSystemStore.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8743 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8743/ ) HADOOP-12508 . delete fails with exception when lease is held on blob. (cnauroth: rev 9e7dcab185abf2fdabb28f2799b9952b5664a4b0) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/SelfRenewingLease.java hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestNativeAzureFileSystemLive.java hadoop-common-project/hadoop-common/CHANGES.txt hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/AzureNativeFileSystemStore.java
          Hide
          cnauroth Chris Nauroth added a comment -

          I have committed this to trunk and branch-2. Gaurav Kanade, thank you for the patch. Xiaoyu Yao, thank you for the code review.

          Show
          cnauroth Chris Nauroth added a comment - I have committed this to trunk and branch-2. Gaurav Kanade , thank you for the patch. Xiaoyu Yao , thank you for the code review.
          Hide
          xyao Xiaoyu Yao added a comment -

          Chris Nauroth, both the main code and the updated unit test looks good to me. +1. Thanks!

          Show
          xyao Xiaoyu Yao added a comment - Chris Nauroth , both the main code and the updated unit test looks good to me. +1. Thanks!
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 5s docker + precommit patch detected.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 3m 0s trunk passed
          +1 compile 0m 12s trunk passed with JDK v1.8.0_60
          +1 compile 0m 13s trunk passed with JDK v1.7.0_79
          +1 checkstyle 0m 7s trunk passed
          +1 mvneclipse 0m 12s trunk passed
          +1 findbugs 0m 31s trunk passed
          +1 javadoc 0m 12s trunk passed with JDK v1.8.0_60
          +1 javadoc 0m 14s trunk passed with JDK v1.7.0_79
          +1 mvninstall 0m 15s the patch passed
          +1 compile 0m 11s the patch passed with JDK v1.8.0_60
          +1 javac 0m 11s the patch passed
          +1 compile 0m 13s the patch passed with JDK v1.7.0_79
          +1 javac 0m 13s the patch passed
          +1 checkstyle 0m 8s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 0m 36s the patch passed
          +1 javadoc 0m 11s the patch passed with JDK v1.8.0_60
          +1 javadoc 0m 14s the patch passed with JDK v1.7.0_79
          +1 unit 0m 59s hadoop-azure in the patch passed with JDK v1.8.0_60.
          +1 unit 1m 12s hadoop-azure in the patch passed with JDK v1.7.0_79.
          +1 asflicense 0m 21s Patch does not generate ASF License warnings.
          10m 17s



          Subsystem Report/Notes
          Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-10-31
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12769960/HADOOP-12508.06.patch
          JIRA Issue HADOOP-12508
          Optional Tests asflicense javac javadoc mvninstall unit findbugs checkstyle compile
          uname Linux 611eece97402 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/patchprocess/apache-yetus-e77b1ce/precommit/personality/hadoop.sh
          git revision trunk / 7fd6416
          Default Java 1.7.0_79
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_60 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79
          findbugs v3.0.0
          JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7989/testReport/
          modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
          Max memory used 226MB
          Powered by Apache Yetus http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7989/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 5s docker + precommit patch detected. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 3m 0s trunk passed +1 compile 0m 12s trunk passed with JDK v1.8.0_60 +1 compile 0m 13s trunk passed with JDK v1.7.0_79 +1 checkstyle 0m 7s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 0m 31s trunk passed +1 javadoc 0m 12s trunk passed with JDK v1.8.0_60 +1 javadoc 0m 14s trunk passed with JDK v1.7.0_79 +1 mvninstall 0m 15s the patch passed +1 compile 0m 11s the patch passed with JDK v1.8.0_60 +1 javac 0m 11s the patch passed +1 compile 0m 13s the patch passed with JDK v1.7.0_79 +1 javac 0m 13s the patch passed +1 checkstyle 0m 8s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 0m 36s the patch passed +1 javadoc 0m 11s the patch passed with JDK v1.8.0_60 +1 javadoc 0m 14s the patch passed with JDK v1.7.0_79 +1 unit 0m 59s hadoop-azure in the patch passed with JDK v1.8.0_60. +1 unit 1m 12s hadoop-azure in the patch passed with JDK v1.7.0_79. +1 asflicense 0m 21s Patch does not generate ASF License warnings. 10m 17s Subsystem Report/Notes Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-10-31 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12769960/HADOOP-12508.06.patch JIRA Issue HADOOP-12508 Optional Tests asflicense javac javadoc mvninstall unit findbugs checkstyle compile uname Linux 611eece97402 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/patchprocess/apache-yetus-e77b1ce/precommit/personality/hadoop.sh git revision trunk / 7fd6416 Default Java 1.7.0_79 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_60 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79 findbugs v3.0.0 JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7989/testReport/ modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure Max memory used 226MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7989/console This message was automatically generated.
          Hide
          cnauroth Chris Nauroth added a comment -

          On further review, I see we don't have the right hooks in place to avoid sleep times in this test completely. Trying to introduce those hooks would be a very intrusive refactoring.

          However, we can still limit our reliance on sleep times and provide a more predictable test by using CountDownLatch. I'm attaching patch v06. The main code is the same, but I've changed the test code to use CountDownLatch for coordination between the 2 threads. There is still some sleep time on the background thread, which is required to make sure we cover the lease acquisition retry logic during the delete. In addition to making the test more predictable, this is also faster. The patch v05 test ran in ~65 seconds. The patch v06 test runs in ~9 seconds.

          Xiaoyu Yao, could we please get your help with a code review on this? I am +1 on the main code portion of this, but I can't commit this based on my own +1, because I've touched the test code portion of the patch. Thanks!

          Show
          cnauroth Chris Nauroth added a comment - On further review, I see we don't have the right hooks in place to avoid sleep times in this test completely. Trying to introduce those hooks would be a very intrusive refactoring. However, we can still limit our reliance on sleep times and provide a more predictable test by using CountDownLatch . I'm attaching patch v06. The main code is the same, but I've changed the test code to use CountDownLatch for coordination between the 2 threads. There is still some sleep time on the background thread, which is required to make sure we cover the lease acquisition retry logic during the delete. In addition to making the test more predictable, this is also faster. The patch v05 test ran in ~65 seconds. The patch v06 test runs in ~9 seconds. Xiaoyu Yao , could we please get your help with a code review on this? I am +1 on the main code portion of this, but I can't commit this based on my own +1, because I've touched the test code portion of the patch. Thanks!
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 5s docker + precommit patch detected.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 2m 53s trunk passed
          +1 compile 0m 13s trunk passed with JDK v1.8.0_60
          +1 compile 0m 13s trunk passed with JDK v1.7.0_79
          +1 checkstyle 0m 9s trunk passed
          +1 mvneclipse 0m 52s trunk passed
          +1 findbugs 0m 32s trunk passed
          +1 javadoc 0m 13s trunk passed with JDK v1.8.0_60
          +1 javadoc 0m 14s trunk passed with JDK v1.7.0_79
          +1 mvninstall 0m 16s the patch passed
          +1 compile 0m 13s the patch passed with JDK v1.8.0_60
          +1 javac 0m 13s the patch passed
          +1 compile 0m 14s the patch passed with JDK v1.7.0_79
          +1 javac 0m 14s the patch passed
          +1 checkstyle 0m 8s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 0m 38s the patch passed
          +1 javadoc 0m 13s the patch passed with JDK v1.8.0_60
          +1 javadoc 0m 15s the patch passed with JDK v1.7.0_79
          +1 unit 1m 4s hadoop-azure in the patch passed with JDK v1.8.0_60.
          +1 unit 1m 12s hadoop-azure in the patch passed with JDK v1.7.0_79.
          +1 asflicense 0m 21s Patch does not generate ASF License warnings.
          11m 6s



          Subsystem Report/Notes
          Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-10-31
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12769922/HADOOP-12508.05.patch
          JIRA Issue HADOOP-12508
          Optional Tests asflicense javac javadoc mvninstall unit findbugs checkstyle compile
          uname Linux b24ca2c27af1 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build@2/patchprocess/apache-yetus-e77b1ce/precommit/personality/hadoop.sh
          git revision trunk / a4a6b5b
          Default Java 1.7.0_79
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_60 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79
          findbugs v3.0.0
          JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7987/testReport/
          modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
          Max memory used 228MB
          Powered by Apache Yetus http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7987/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 5s docker + precommit patch detected. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 2m 53s trunk passed +1 compile 0m 13s trunk passed with JDK v1.8.0_60 +1 compile 0m 13s trunk passed with JDK v1.7.0_79 +1 checkstyle 0m 9s trunk passed +1 mvneclipse 0m 52s trunk passed +1 findbugs 0m 32s trunk passed +1 javadoc 0m 13s trunk passed with JDK v1.8.0_60 +1 javadoc 0m 14s trunk passed with JDK v1.7.0_79 +1 mvninstall 0m 16s the patch passed +1 compile 0m 13s the patch passed with JDK v1.8.0_60 +1 javac 0m 13s the patch passed +1 compile 0m 14s the patch passed with JDK v1.7.0_79 +1 javac 0m 14s the patch passed +1 checkstyle 0m 8s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 0m 38s the patch passed +1 javadoc 0m 13s the patch passed with JDK v1.8.0_60 +1 javadoc 0m 15s the patch passed with JDK v1.7.0_79 +1 unit 1m 4s hadoop-azure in the patch passed with JDK v1.8.0_60. +1 unit 1m 12s hadoop-azure in the patch passed with JDK v1.7.0_79. +1 asflicense 0m 21s Patch does not generate ASF License warnings. 11m 6s Subsystem Report/Notes Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-10-31 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12769922/HADOOP-12508.05.patch JIRA Issue HADOOP-12508 Optional Tests asflicense javac javadoc mvninstall unit findbugs checkstyle compile uname Linux b24ca2c27af1 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build@2/patchprocess/apache-yetus-e77b1ce/precommit/personality/hadoop.sh git revision trunk / a4a6b5b Default Java 1.7.0_79 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_60 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79 findbugs v3.0.0 JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7987/testReport/ modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure Max memory used 228MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7987/console This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 9s docker + precommit patch detected.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 3m 48s trunk passed
          +1 compile 0m 16s trunk passed with JDK v1.8.0_60
          +1 compile 0m 18s trunk passed with JDK v1.7.0_79
          +1 checkstyle 0m 10s trunk passed
          +1 mvneclipse 0m 15s trunk passed
          +1 findbugs 0m 38s trunk passed
          +1 javadoc 0m 17s trunk passed with JDK v1.8.0_60
          +1 javadoc 0m 17s trunk passed with JDK v1.7.0_79
          +1 mvninstall 0m 26s the patch passed
          +1 compile 0m 17s the patch passed with JDK v1.8.0_60
          +1 javac 0m 17s the patch passed
          +1 compile 0m 16s the patch passed with JDK v1.7.0_79
          +1 javac 0m 16s the patch passed
          +1 checkstyle 0m 10s the patch passed
          +1 mvneclipse 0m 14s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 0m 49s the patch passed
          +1 javadoc 0m 17s the patch passed with JDK v1.8.0_60
          +1 javadoc 0m 19s the patch passed with JDK v1.7.0_79
          +1 unit 1m 29s hadoop-azure in the patch passed with JDK v1.8.0_60.
          +1 unit 1m 30s hadoop-azure in the patch passed with JDK v1.7.0_79.
          +1 asflicense 0m 26s Patch does not generate ASF License warnings.
          13m 34s



          Subsystem Report/Notes
          Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-10-31
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12769922/HADOOP-12508.05.patch
          JIRA Issue HADOOP-12508
          Optional Tests asflicense javac javadoc mvninstall unit findbugs checkstyle compile
          uname Linux 22839e93677b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/patchprocess/apache-yetus-e77b1ce/precommit/personality/hadoop.sh
          git revision trunk / a4a6b5b
          Default Java 1.7.0_79
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_60 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79
          findbugs v3.0.0
          JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7984/testReport/
          modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
          Max memory used 227MB
          Powered by Apache Yetus http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7984/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 9s docker + precommit patch detected. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 3m 48s trunk passed +1 compile 0m 16s trunk passed with JDK v1.8.0_60 +1 compile 0m 18s trunk passed with JDK v1.7.0_79 +1 checkstyle 0m 10s trunk passed +1 mvneclipse 0m 15s trunk passed +1 findbugs 0m 38s trunk passed +1 javadoc 0m 17s trunk passed with JDK v1.8.0_60 +1 javadoc 0m 17s trunk passed with JDK v1.7.0_79 +1 mvninstall 0m 26s the patch passed +1 compile 0m 17s the patch passed with JDK v1.8.0_60 +1 javac 0m 17s the patch passed +1 compile 0m 16s the patch passed with JDK v1.7.0_79 +1 javac 0m 16s the patch passed +1 checkstyle 0m 10s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 0m 49s the patch passed +1 javadoc 0m 17s the patch passed with JDK v1.8.0_60 +1 javadoc 0m 19s the patch passed with JDK v1.7.0_79 +1 unit 1m 29s hadoop-azure in the patch passed with JDK v1.8.0_60. +1 unit 1m 30s hadoop-azure in the patch passed with JDK v1.7.0_79. +1 asflicense 0m 26s Patch does not generate ASF License warnings. 13m 34s Subsystem Report/Notes Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-10-31 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12769922/HADOOP-12508.05.patch JIRA Issue HADOOP-12508 Optional Tests asflicense javac javadoc mvninstall unit findbugs checkstyle compile uname Linux 22839e93677b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/patchprocess/apache-yetus-e77b1ce/precommit/personality/hadoop.sh git revision trunk / a4a6b5b Default Java 1.7.0_79 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_60 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79 findbugs v3.0.0 JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7984/testReport/ modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure Max memory used 227MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7984/console This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 13s docker + precommit patch detected.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 4m 3s trunk passed
          +1 compile 0m 18s trunk passed with JDK v1.8.0_60
          +1 compile 0m 18s trunk passed with JDK v1.7.0_79
          +1 checkstyle 0m 11s trunk passed
          +1 mvneclipse 0m 17s trunk passed
          +1 findbugs 0m 37s trunk passed
          +1 javadoc 0m 16s trunk passed with JDK v1.8.0_60
          +1 javadoc 0m 17s trunk passed with JDK v1.7.0_79
          +1 mvninstall 0m 19s the patch passed
          +1 compile 0m 18s the patch passed with JDK v1.8.0_60
          +1 javac 0m 18s the patch passed
          +1 compile 0m 17s the patch passed with JDK v1.7.0_79
          +1 javac 0m 17s the patch passed
          +1 checkstyle 0m 10s the patch passed
          +1 mvneclipse 0m 15s the patch passed
          -1 whitespace 0m 0s The patch has 1 line(s) with tabs.
          +1 findbugs 0m 49s the patch passed
          +1 javadoc 0m 17s the patch passed with JDK v1.8.0_60
          +1 javadoc 0m 19s the patch passed with JDK v1.7.0_79
          +1 unit 1m 31s hadoop-azure in the patch passed with JDK v1.8.0_60.
          +1 unit 1m 33s hadoop-azure in the patch passed with JDK v1.7.0_79.
          +1 asflicense 0m 30s Patch does not generate ASF License warnings.
          14m 4s



          Subsystem Report/Notes
          Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-10-31
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12769898/HADOOP-12508.05.patch
          JIRA Issue HADOOP-12508
          Optional Tests asflicense javac javadoc mvninstall unit findbugs checkstyle compile
          uname Linux f7d6208a97ac 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/patchprocess/apache-yetus-e77b1ce/precommit/personality/hadoop.sh
          git revision trunk / a4a6b5b
          Default Java 1.7.0_79
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_60 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79
          findbugs v3.0.0
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/7983/artifact/patchprocess/whitespace-tabs.txt
          JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7983/testReport/
          modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
          Max memory used 230MB
          Powered by Apache Yetus http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7983/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s docker + precommit patch detected. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 4m 3s trunk passed +1 compile 0m 18s trunk passed with JDK v1.8.0_60 +1 compile 0m 18s trunk passed with JDK v1.7.0_79 +1 checkstyle 0m 11s trunk passed +1 mvneclipse 0m 17s trunk passed +1 findbugs 0m 37s trunk passed +1 javadoc 0m 16s trunk passed with JDK v1.8.0_60 +1 javadoc 0m 17s trunk passed with JDK v1.7.0_79 +1 mvninstall 0m 19s the patch passed +1 compile 0m 18s the patch passed with JDK v1.8.0_60 +1 javac 0m 18s the patch passed +1 compile 0m 17s the patch passed with JDK v1.7.0_79 +1 javac 0m 17s the patch passed +1 checkstyle 0m 10s the patch passed +1 mvneclipse 0m 15s the patch passed -1 whitespace 0m 0s The patch has 1 line(s) with tabs. +1 findbugs 0m 49s the patch passed +1 javadoc 0m 17s the patch passed with JDK v1.8.0_60 +1 javadoc 0m 19s the patch passed with JDK v1.7.0_79 +1 unit 1m 31s hadoop-azure in the patch passed with JDK v1.8.0_60. +1 unit 1m 33s hadoop-azure in the patch passed with JDK v1.7.0_79. +1 asflicense 0m 30s Patch does not generate ASF License warnings. 14m 4s Subsystem Report/Notes Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-10-31 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12769898/HADOOP-12508.05.patch JIRA Issue HADOOP-12508 Optional Tests asflicense javac javadoc mvninstall unit findbugs checkstyle compile uname Linux f7d6208a97ac 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/patchprocess/apache-yetus-e77b1ce/precommit/personality/hadoop.sh git revision trunk / a4a6b5b Default Java 1.7.0_79 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_60 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79 findbugs v3.0.0 whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/7983/artifact/patchprocess/whitespace-tabs.txt JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7983/testReport/ modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure Max memory used 230MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7983/console This message was automatically generated.
          Hide
          gouravk Gaurav Kanade added a comment -

          Chris Nauroth - I have rebased and fixed the tabs issue, but I don't see a Jenkins run yet so still waiting for validation.

          FOr the issue of test using hard-coded values to coordinate two threads, I believe I can work around this by having the original thread acquire the lease and then attempt a delete. IN the interval between these two steps it will kick off background thread which will merely wait for a fixed time and then release the lease. This should essentially simulate the situation; I have already verified this test fails without the patch and passes after

          Show
          gouravk Gaurav Kanade added a comment - Chris Nauroth - I have rebased and fixed the tabs issue, but I don't see a Jenkins run yet so still waiting for validation. FOr the issue of test using hard-coded values to coordinate two threads, I believe I can work around this by having the original thread acquire the lease and then attempt a delete. IN the interval between these two steps it will kick off background thread which will merely wait for a fixed time and then release the lease. This should essentially simulate the situation; I have already verified this test fails without the patch and passes after
          Hide
          gouravk Gaurav Kanade added a comment -

          Thanks Chris Nauroth, for the check I rebased and fixed the tabs issue. Will need to work through the sync issue.

          Show
          gouravk Gaurav Kanade added a comment - Thanks Chris Nauroth, for the check I rebased and fixed the tabs issue. Will need to work through the sync issue.
          Hide
          cnauroth Chris Nauroth added a comment -

          Assuming the patch gets rebased, but stays basically the same:

          1. I can tell it's going to fail the pre-commit checks because of indentation. Please consistently indent by 2 spaces, not tabs.
          2. Using hard-coded sleep times to coordinate 2 threads is non-deterministic. This will be prone to intermittent spurious test failures. A good alternative is CountDownLatch, which allows you to block one thread waiting for another thread to complete its work and then signal.
          Show
          cnauroth Chris Nauroth added a comment - Assuming the patch gets rebased, but stays basically the same: I can tell it's going to fail the pre-commit checks because of indentation. Please consistently indent by 2 spaces, not tabs. Using hard-coded sleep times to coordinate 2 threads is non-deterministic. This will be prone to intermittent spurious test failures. A good alternative is CountDownLatch , which allows you to block one thread waiting for another thread to complete its work and then signal.
          Hide
          cnauroth Chris Nauroth added a comment -

          Gaurav Kanade, the latest patch does not apply to trunk. Can you please check if the patch needs to be rebased? Thank you.

          Show
          cnauroth Chris Nauroth added a comment - Gaurav Kanade , the latest patch does not apply to trunk. Can you please check if the patch needs to be rebased? Thank you.
          Hide
          gouravk Gaurav Kanade added a comment -

          Re-submitting the patch after addressing issues related to test coverage. Now I have included a unit test that should fail before the patch and should pass after. Also verified this locally via custom built jar

          Show
          gouravk Gaurav Kanade added a comment - Re-submitting the patch after addressing issues related to test coverage. Now I have included a unit test that should fail before the patch and should pass after. Also verified this locally via custom built jar
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 6s docker + precommit patch detected.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 8m 54s trunk passed
          +1 compile 0m 13s trunk passed with JDK v1.8.0_66
          +1 compile 0m 14s trunk passed with JDK v1.7.0_79
          +1 checkstyle 0m 11s trunk passed
          +1 mvneclipse 0m 50s trunk passed
          +1 findbugs 0m 37s trunk passed
          +1 javadoc 0m 15s trunk passed with JDK v1.8.0_66
          +1 javadoc 0m 14s trunk passed with JDK v1.7.0_79
          +1 mvninstall 0m 15s the patch passed
          +1 compile 0m 12s the patch passed with JDK v1.8.0_66
          +1 javac 0m 12s the patch passed
          +1 compile 0m 14s the patch passed with JDK v1.7.0_79
          +1 javac 0m 14s the patch passed
          +1 checkstyle 0m 9s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          -1 whitespace 0m 0s The patch has 3 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 findbugs 0m 39s the patch passed
          +1 javadoc 0m 13s the patch passed with JDK v1.8.0_66
          +1 javadoc 0m 14s the patch passed with JDK v1.7.0_79
          +1 unit 1m 8s hadoop-azure in the patch passed with JDK v1.8.0_66.
          +1 unit 1m 17s hadoop-azure in the patch passed with JDK v1.7.0_79.
          -1 asflicense 0m 23s Patch generated 1 ASF License warnings.
          17m 28s



          Subsystem Report/Notes
          Docker Client=1.7.0 Server=1.7.0 Image:test-patch-base-hadoop-date2015-10-27
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12768829/HADOOP-12508.03.patch
          JIRA Issue HADOOP-12508
          Optional Tests asflicense javac javadoc mvninstall unit findbugs checkstyle compile
          uname Linux 7384905fbce0 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/patchprocess/apache-yetus-b9c369f/dev-support/personality/hadoop.sh
          git revision trunk / 56e4f62
          Default Java 1.7.0_79
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79
          findbugs v3.0.0
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/7942/artifact/patchprocess/whitespace-eol.txt
          JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7942/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HADOOP-Build/7942/artifact/patchprocess/patch-asflicense-problems.txt
          Max memory used 227MB
          Powered by Apache Yetus http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7942/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 6s docker + precommit patch detected. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 8m 54s trunk passed +1 compile 0m 13s trunk passed with JDK v1.8.0_66 +1 compile 0m 14s trunk passed with JDK v1.7.0_79 +1 checkstyle 0m 11s trunk passed +1 mvneclipse 0m 50s trunk passed +1 findbugs 0m 37s trunk passed +1 javadoc 0m 15s trunk passed with JDK v1.8.0_66 +1 javadoc 0m 14s trunk passed with JDK v1.7.0_79 +1 mvninstall 0m 15s the patch passed +1 compile 0m 12s the patch passed with JDK v1.8.0_66 +1 javac 0m 12s the patch passed +1 compile 0m 14s the patch passed with JDK v1.7.0_79 +1 javac 0m 14s the patch passed +1 checkstyle 0m 9s the patch passed +1 mvneclipse 0m 11s the patch passed -1 whitespace 0m 0s The patch has 3 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 findbugs 0m 39s the patch passed +1 javadoc 0m 13s the patch passed with JDK v1.8.0_66 +1 javadoc 0m 14s the patch passed with JDK v1.7.0_79 +1 unit 1m 8s hadoop-azure in the patch passed with JDK v1.8.0_66. +1 unit 1m 17s hadoop-azure in the patch passed with JDK v1.7.0_79. -1 asflicense 0m 23s Patch generated 1 ASF License warnings. 17m 28s Subsystem Report/Notes Docker Client=1.7.0 Server=1.7.0 Image:test-patch-base-hadoop-date2015-10-27 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12768829/HADOOP-12508.03.patch JIRA Issue HADOOP-12508 Optional Tests asflicense javac javadoc mvninstall unit findbugs checkstyle compile uname Linux 7384905fbce0 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/patchprocess/apache-yetus-b9c369f/dev-support/personality/hadoop.sh git revision trunk / 56e4f62 Default Java 1.7.0_79 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79 findbugs v3.0.0 whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/7942/artifact/patchprocess/whitespace-eol.txt JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7942/testReport/ asflicense https://builds.apache.org/job/PreCommit-HADOOP-Build/7942/artifact/patchprocess/patch-asflicense-problems.txt Max memory used 227MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7942/console This message was automatically generated.
          Hide
          cnauroth Chris Nauroth added a comment -

          Gaurav Kanade, thank you for adding the test, but this doesn't really cover execution of the new logic. (The test passes with and without the main code change applied.) I think the idea behind the test could be made to work if the lease was acquired in a background thread, and then the main JUnit thread attempted the delete. We'd then expect the new logic to wait on its own lease acquisition and eventually complete the delete.

          Show
          cnauroth Chris Nauroth added a comment - Gaurav Kanade , thank you for adding the test, but this doesn't really cover execution of the new logic. (The test passes with and without the main code change applied.) I think the idea behind the test could be made to work if the lease was acquired in a background thread, and then the main JUnit thread attempted the delete. We'd then expect the new logic to wait on its own lease acquisition and eventually complete the delete.
          Hide
          gouravk Gaurav Kanade added a comment -

          Chris Nauroth

          Yes this could be potentially an improvement to break the lease explicitly. As of now I have tried to follow the principle of minimum disruption to existing code. As can be seen we never hit this scenario until this new HDP 2.3 test was written though the scenario was valid since the beginning. So it might be pretty rare case.

          I believe in the light of the recent weeks experience we plan to take a re-look at the WASB driver code from point of view of robustness to concurrency process and test coverage; at that time we can revisit this issue.

          Meanwhile on the test front I am planning to write a test that at least partially validates this patch - i.e. tests that we are catching the exception appropriately and submitting a new patch shortly. We still will not be able to test the handling part as this will require extra infra.

          Adding Duo Xu, Pravin Mittal, Lin Chan for their thoughts on your suggestion.

          Show
          gouravk Gaurav Kanade added a comment - Chris Nauroth Yes this could be potentially an improvement to break the lease explicitly. As of now I have tried to follow the principle of minimum disruption to existing code. As can be seen we never hit this scenario until this new HDP 2.3 test was written though the scenario was valid since the beginning. So it might be pretty rare case. I believe in the light of the recent weeks experience we plan to take a re-look at the WASB driver code from point of view of robustness to concurrency process and test coverage; at that time we can revisit this issue. Meanwhile on the test front I am planning to write a test that at least partially validates this patch - i.e. tests that we are catching the exception appropriately and submitting a new patch shortly. We still will not be able to test the handling part as this will require extra infra. Adding Duo Xu , Pravin Mittal , Lin Chan for their thoughts on your suggestion.
          Hide
          cnauroth Chris Nauroth added a comment -

          Thank you for the further details, Gaurav Kanade.

          It sounds like the root cause here is likely a difference in the lease semantics of WASB compared to HDFS. In HDFS, a delete operation succeeds despite another concurrent client holding a lease on that path. In WASB, it fails due to the lease enforcement of the backing Azure Storage.

          Would it be a potential improvement to break the lease explicitly? The downside of the current patch is that a client calling delete on a path with a lease held and the leasing process still alive will suffer up to 60 seconds additional latency. If the lease was broken explicitly, then you could send a shorter x-ms-lease-break-period to avoid that latency.

          Show
          cnauroth Chris Nauroth added a comment - Thank you for the further details, Gaurav Kanade . It sounds like the root cause here is likely a difference in the lease semantics of WASB compared to HDFS. In HDFS, a delete operation succeeds despite another concurrent client holding a lease on that path. In WASB, it fails due to the lease enforcement of the backing Azure Storage. Would it be a potential improvement to break the lease explicitly? The downside of the current patch is that a client calling delete on a path with a lease held and the leasing process still alive will suffer up to 60 seconds additional latency. If the lease was broken explicitly, then you could send a shorter x-ms-lease-break-period to avoid that latency.
          Hide
          gouravk Gaurav Kanade added a comment -

          Hey Chris Nauroth

          Thanks for the review! The acquireLease makes a call to obtain a SelfRenewingLease. If you look at the SelfRenewingLease class it will keep waiting to acquire lease until it finally acquires it. The concurrent process that holds the lease is dead - so once its existing hold on the lease expires it will not self renew (as the self-renewing thread will be dead as well). This patch seeks to address this particular condition. Thus the expected behavior would be the process that died while still holding a dangling lease will not attempt to renew. The process will keep on trying to acquire the lease until it gets it which will be once the existing lease expires (The default lease holding time is 60 sec, so it will be a worst case test for 60 sec). At a bare minimum this patch will not break anything that is already broken, and it will expose a deeper issue if exists.

          As for the testing, we are working on designing a framework that can test error conditions caused by concurrent processes exiting unexpectedly (this seems to be the class of issues we are hitting and are exposed by the new HBase test introduced in HDP 2.3 - these seem to be rarely occurring in practice as no customer seems to have hit them yet). In the meanwhile if you have ideas regarding the kind of testing that can be done quickly in the short term would love to hear those.

          Show
          gouravk Gaurav Kanade added a comment - Hey Chris Nauroth Thanks for the review! The acquireLease makes a call to obtain a SelfRenewingLease. If you look at the SelfRenewingLease class it will keep waiting to acquire lease until it finally acquires it. The concurrent process that holds the lease is dead - so once its existing hold on the lease expires it will not self renew (as the self-renewing thread will be dead as well). This patch seeks to address this particular condition. Thus the expected behavior would be the process that died while still holding a dangling lease will not attempt to renew. The process will keep on trying to acquire the lease until it gets it which will be once the existing lease expires (The default lease holding time is 60 sec, so it will be a worst case test for 60 sec). At a bare minimum this patch will not break anything that is already broken, and it will expose a deeper issue if exists. As for the testing, we are working on designing a framework that can test error conditions caused by concurrent processes exiting unexpectedly (this seems to be the class of issues we are hitting and are exposed by the new HBase test introduced in HDP 2.3 - these seem to be rarely occurring in practice as no customer seems to have hit them yet). In the meanwhile if you have ideas regarding the kind of testing that can be done quickly in the short term would love to hear those.
          Hide
          cnauroth Chris Nauroth added a comment -

          Hello Gaurav Kanade.

          Thank you for posting this patch. However, I don't see how this could really fix the problem. If a delete operation is failing because of another concurrent process holding a lease, then I expect simply calling acquireLease immediately would fail again, because it's impossible for 2 clients to hold a lease on the same blob at the same time. Perhaps it would appear to work sometimes if sufficient time passed for the other process to release its lease, but this would be a non-deterministic race condition.

          Has the patch been tested?

          Show
          cnauroth Chris Nauroth added a comment - Hello Gaurav Kanade . Thank you for posting this patch. However, I don't see how this could really fix the problem. If a delete operation is failing because of another concurrent process holding a lease, then I expect simply calling acquireLease immediately would fail again, because it's impossible for 2 clients to hold a lease on the same blob at the same time. Perhaps it would appear to work sometimes if sufficient time passed for the other process to release its lease, but this would be a non-deterministic race condition. Has the patch been tested?
          Hide
          gouravk Gaurav Kanade added a comment -
          Show
          gouravk Gaurav Kanade added a comment - Chris Nauroth , Duo Xu , Lin Chan , Pravin Mittal , Dushyanth - please take a look
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 29s docker + precommit patch detected.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 4m 12s trunk passed
          +1 compile 0m 21s trunk passed with JDK v1.8.0_60
          +1 compile 0m 20s trunk passed with JDK v1.7.0_79
          +1 checkstyle 0m 11s trunk passed
          +1 mvneclipse 0m 17s trunk passed
          +1 findbugs 0m 38s trunk passed
          +1 javadoc 0m 22s trunk passed with JDK v1.8.0_60
          +1 javadoc 0m 20s trunk passed with JDK v1.7.0_79
          +1 mvninstall 0m 22s the patch passed
          +1 compile 0m 17s the patch passed with JDK v1.8.0_60
          +1 javac 0m 17s the patch passed
          +1 compile 0m 16s the patch passed with JDK v1.7.0_79
          +1 javac 0m 16s the patch passed
          +1 checkstyle 0m 9s the patch passed
          +1 mvneclipse 0m 13s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 0m 47s the patch passed
          +1 javadoc 0m 16s the patch passed with JDK v1.8.0_60
          +1 javadoc 0m 17s the patch passed with JDK v1.7.0_79
          +1 unit 1m 23s hadoop-azure in the patch passed with JDK v1.8.0_60.
          +1 unit 1m 28s hadoop-azure in the patch passed with JDK v1.7.0_79.
          -1 asflicense 0m 27s Patch generated 1 ASF License warnings.
          14m 25s



          Subsystem Report/Notes
          Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-10-25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12768633/HADOOP-12508.02.patch
          JIRA Issue HADOOP-12508
          Optional Tests asflicense javac javadoc mvninstall unit findbugs checkstyle compile
          uname Linux 03d400cfd0dc 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/patchprocess/apache-yetus-b9c369f/dev-support/personality/hadoop.sh
          git revision trunk / ab8eb87
          Default Java 1.7.0_79
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_60 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79
          findbugs v3.0.0
          JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7930/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HADOOP-Build/7930/artifact/patchprocess/patch-asflicense-problems.txt
          Max memory used 227MB
          Powered by Apache Yetus http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7930/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 29s docker + precommit patch detected. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 4m 12s trunk passed +1 compile 0m 21s trunk passed with JDK v1.8.0_60 +1 compile 0m 20s trunk passed with JDK v1.7.0_79 +1 checkstyle 0m 11s trunk passed +1 mvneclipse 0m 17s trunk passed +1 findbugs 0m 38s trunk passed +1 javadoc 0m 22s trunk passed with JDK v1.8.0_60 +1 javadoc 0m 20s trunk passed with JDK v1.7.0_79 +1 mvninstall 0m 22s the patch passed +1 compile 0m 17s the patch passed with JDK v1.8.0_60 +1 javac 0m 17s the patch passed +1 compile 0m 16s the patch passed with JDK v1.7.0_79 +1 javac 0m 16s the patch passed +1 checkstyle 0m 9s the patch passed +1 mvneclipse 0m 13s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 0m 47s the patch passed +1 javadoc 0m 16s the patch passed with JDK v1.8.0_60 +1 javadoc 0m 17s the patch passed with JDK v1.7.0_79 +1 unit 1m 23s hadoop-azure in the patch passed with JDK v1.8.0_60. +1 unit 1m 28s hadoop-azure in the patch passed with JDK v1.7.0_79. -1 asflicense 0m 27s Patch generated 1 ASF License warnings. 14m 25s Subsystem Report/Notes Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-10-25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12768633/HADOOP-12508.02.patch JIRA Issue HADOOP-12508 Optional Tests asflicense javac javadoc mvninstall unit findbugs checkstyle compile uname Linux 03d400cfd0dc 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/patchprocess/apache-yetus-b9c369f/dev-support/personality/hadoop.sh git revision trunk / ab8eb87 Default Java 1.7.0_79 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_60 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79 findbugs v3.0.0 JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7930/testReport/ asflicense https://builds.apache.org/job/PreCommit-HADOOP-Build/7930/artifact/patchprocess/patch-asflicense-problems.txt Max memory used 227MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7930/console This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 7s docker + precommit patch detected.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 3m 10s trunk passed
          +1 compile 0m 12s trunk passed with JDK v1.8.0_60
          +1 compile 0m 14s trunk passed with JDK v1.7.0_79
          +1 checkstyle 0m 7s trunk passed
          +1 mvneclipse 0m 14s trunk passed
          +1 findbugs 0m 28s trunk passed
          +1 javadoc 0m 12s trunk passed with JDK v1.8.0_60
          +1 javadoc 0m 14s trunk passed with JDK v1.7.0_79
          +1 mvninstall 0m 15s the patch passed
          +1 compile 0m 11s the patch passed with JDK v1.8.0_60
          +1 javac 0m 11s the patch passed
          +1 compile 0m 14s the patch passed with JDK v1.7.0_79
          +1 javac 0m 14s the patch passed
          -1 checkstyle 0m 7s Patch generated 1 new checkstyle issues in hadoop-tools/hadoop-azure (total was 31, now 32).
          +1 mvneclipse 0m 12s the patch passed
          -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 findbugs 0m 37s the patch passed
          +1 javadoc 0m 12s the patch passed with JDK v1.8.0_60
          +1 javadoc 0m 14s the patch passed with JDK v1.7.0_79
          +1 unit 0m 58s hadoop-azure in the patch passed with JDK v1.8.0_60.
          +1 unit 1m 11s hadoop-azure in the patch passed with JDK v1.7.0_79.
          -1 asflicense 0m 23s Patch generated 1 ASF License warnings.
          10m 30s



          Subsystem Report/Notes
          Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-10-25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12768624/HADOOP-12508.01.patch
          JIRA Issue HADOOP-12508
          Optional Tests asflicense javac javadoc mvninstall unit findbugs checkstyle compile
          uname Linux cd5c514665a0 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/patchprocess/apache-yetus-b9c369f/dev-support/personality/hadoop.sh
          git revision trunk / ab8eb87
          Default Java 1.7.0_79
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_60 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/7929/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-azure.txt
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/7929/artifact/patchprocess/whitespace-eol.txt
          JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7929/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HADOOP-Build/7929/artifact/patchprocess/patch-asflicense-problems.txt
          Max memory used 229MB
          Powered by Apache Yetus http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7929/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 7s docker + precommit patch detected. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 3m 10s trunk passed +1 compile 0m 12s trunk passed with JDK v1.8.0_60 +1 compile 0m 14s trunk passed with JDK v1.7.0_79 +1 checkstyle 0m 7s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 0m 28s trunk passed +1 javadoc 0m 12s trunk passed with JDK v1.8.0_60 +1 javadoc 0m 14s trunk passed with JDK v1.7.0_79 +1 mvninstall 0m 15s the patch passed +1 compile 0m 11s the patch passed with JDK v1.8.0_60 +1 javac 0m 11s the patch passed +1 compile 0m 14s the patch passed with JDK v1.7.0_79 +1 javac 0m 14s the patch passed -1 checkstyle 0m 7s Patch generated 1 new checkstyle issues in hadoop-tools/hadoop-azure (total was 31, now 32). +1 mvneclipse 0m 12s the patch passed -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 findbugs 0m 37s the patch passed +1 javadoc 0m 12s the patch passed with JDK v1.8.0_60 +1 javadoc 0m 14s the patch passed with JDK v1.7.0_79 +1 unit 0m 58s hadoop-azure in the patch passed with JDK v1.8.0_60. +1 unit 1m 11s hadoop-azure in the patch passed with JDK v1.7.0_79. -1 asflicense 0m 23s Patch generated 1 ASF License warnings. 10m 30s Subsystem Report/Notes Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-10-25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12768624/HADOOP-12508.01.patch JIRA Issue HADOOP-12508 Optional Tests asflicense javac javadoc mvninstall unit findbugs checkstyle compile uname Linux cd5c514665a0 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/patchprocess/apache-yetus-b9c369f/dev-support/personality/hadoop.sh git revision trunk / ab8eb87 Default Java 1.7.0_79 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_60 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/7929/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-azure.txt whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/7929/artifact/patchprocess/whitespace-eol.txt JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7929/testReport/ asflicense https://builds.apache.org/job/PreCommit-HADOOP-Build/7929/artifact/patchprocess/patch-asflicense-problems.txt Max memory used 229MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7929/console This message was automatically generated.

            People

            • Assignee:
              gouravk Gaurav Kanade
              Reporter:
              gouravk Gaurav Kanade
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Due:
                Created:
                Updated:
                Resolved:

                Development