Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-12540

TestAzureFileSystemInstrumentation#testClientErrorMetrics fails intermittently due to assumption that a lease error will be thrown.

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: fs/azure, test
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      HADOOP-12508 changed the behavior of an Azure Storage lease violation during deletes. It appears that TestAzureFileSystemInstrumentation#testClientErrorMetrics is partly dependent on the old behavior for simulating an error to be tracked by the metrics system. I am seeing intermittent failures in this test.

      1. HADOOP-12540.03.patch
        2 kB
        Gaurav Kanade
      2. HADOOP-12540.02.patch
        2 kB
        Gaurav Kanade
      3. HADOOP-12540.01.patch
        5 kB
        Gaurav Kanade

        Issue Links

          Activity

          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Hdfs-trunk #2510 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2510/)
          HADOOP-12540. TestAzureFileSystemInstrumentation#testClientErrorMetrics (cnauroth: rev 0fb1867fd62b5df664ad66386d6067db8fbf2317)

          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/metrics/TestAzureFileSystemInstrumentation.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk #2510 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2510/ ) HADOOP-12540 . TestAzureFileSystemInstrumentation#testClientErrorMetrics (cnauroth: rev 0fb1867fd62b5df664ad66386d6067db8fbf2317) hadoop-common-project/hadoop-common/CHANGES.txt hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/metrics/TestAzureFileSystemInstrumentation.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #572 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/572/)
          HADOOP-12540. TestAzureFileSystemInstrumentation#testClientErrorMetrics (cnauroth: rev 0fb1867fd62b5df664ad66386d6067db8fbf2317)

          • hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/metrics/TestAzureFileSystemInstrumentation.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #572 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/572/ ) HADOOP-12540 . TestAzureFileSystemInstrumentation#testClientErrorMetrics (cnauroth: rev 0fb1867fd62b5df664ad66386d6067db8fbf2317) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/metrics/TestAzureFileSystemInstrumentation.java hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #628 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/628/)
          HADOOP-12540. TestAzureFileSystemInstrumentation#testClientErrorMetrics (cnauroth: rev 0fb1867fd62b5df664ad66386d6067db8fbf2317)

          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/metrics/TestAzureFileSystemInstrumentation.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #628 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/628/ ) HADOOP-12540 . TestAzureFileSystemInstrumentation#testClientErrorMetrics (cnauroth: rev 0fb1867fd62b5df664ad66386d6067db8fbf2317) hadoop-common-project/hadoop-common/CHANGES.txt hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/metrics/TestAzureFileSystemInstrumentation.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Yarn-trunk #1362 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1362/)
          HADOOP-12540. TestAzureFileSystemInstrumentation#testClientErrorMetrics (cnauroth: rev 0fb1867fd62b5df664ad66386d6067db8fbf2317)

          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/metrics/TestAzureFileSystemInstrumentation.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #1362 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1362/ ) HADOOP-12540 . TestAzureFileSystemInstrumentation#testClientErrorMetrics (cnauroth: rev 0fb1867fd62b5df664ad66386d6067db8fbf2317) hadoop-common-project/hadoop-common/CHANGES.txt hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/metrics/TestAzureFileSystemInstrumentation.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2569 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2569/)
          HADOOP-12540. TestAzureFileSystemInstrumentation#testClientErrorMetrics (cnauroth: rev 0fb1867fd62b5df664ad66386d6067db8fbf2317)

          • hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/metrics/TestAzureFileSystemInstrumentation.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2569 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2569/ ) HADOOP-12540 . TestAzureFileSystemInstrumentation#testClientErrorMetrics (cnauroth: rev 0fb1867fd62b5df664ad66386d6067db8fbf2317) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/metrics/TestAzureFileSystemInstrumentation.java hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #639 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/639/)
          HADOOP-12540. TestAzureFileSystemInstrumentation#testClientErrorMetrics (cnauroth: rev 0fb1867fd62b5df664ad66386d6067db8fbf2317)

          • hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/metrics/TestAzureFileSystemInstrumentation.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #639 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/639/ ) HADOOP-12540 . TestAzureFileSystemInstrumentation#testClientErrorMetrics (cnauroth: rev 0fb1867fd62b5df664ad66386d6067db8fbf2317) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/metrics/TestAzureFileSystemInstrumentation.java hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #8757 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8757/)
          HADOOP-12540. TestAzureFileSystemInstrumentation#testClientErrorMetrics (cnauroth: rev 0fb1867fd62b5df664ad66386d6067db8fbf2317)

          • hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/metrics/TestAzureFileSystemInstrumentation.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8757 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8757/ ) HADOOP-12540 . TestAzureFileSystemInstrumentation#testClientErrorMetrics (cnauroth: rev 0fb1867fd62b5df664ad66386d6067db8fbf2317) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/metrics/TestAzureFileSystemInstrumentation.java hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          cnauroth Chris Nauroth added a comment -

          I have committed this to trunk and branch-2. Gaurav, thank you for contributing the patch.

          Show
          cnauroth Chris Nauroth added a comment - I have committed this to trunk and branch-2. Gaurav, thank you for contributing the patch.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 14s docker + precommit patch detected.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 3m 20s trunk passed
          +1 compile 0m 15s trunk passed with JDK v1.8.0_66
          +1 compile 0m 14s trunk passed with JDK v1.7.0_79
          +1 checkstyle 0m 9s trunk passed
          +1 mvneclipse 0m 16s trunk passed
          +1 findbugs 0m 33s trunk passed
          +1 javadoc 0m 13s trunk passed with JDK v1.8.0_66
          +1 javadoc 0m 15s trunk passed with JDK v1.7.0_79
          +1 mvninstall 0m 17s the patch passed
          +1 compile 0m 13s the patch passed with JDK v1.8.0_66
          +1 javac 0m 13s the patch passed
          +1 compile 0m 15s the patch passed with JDK v1.7.0_79
          +1 javac 0m 15s the patch passed
          +1 checkstyle 0m 8s the patch passed
          +1 mvneclipse 0m 13s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 0m 41s the patch passed
          +1 javadoc 0m 15s the patch passed with JDK v1.8.0_66
          +1 javadoc 0m 15s the patch passed with JDK v1.7.0_79
          +1 unit 1m 7s hadoop-azure in the patch passed with JDK v1.8.0_66.
          +1 unit 1m 21s hadoop-azure in the patch passed with JDK v1.7.0_79.
          +1 asflicense 0m 29s Patch does not generate ASF License warnings.
          11m 46s



          Subsystem Report/Notes
          Docker Client=1.7.0 Server=1.7.0 Image:test-patch-base-hadoop-date2015-11-04
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12770460/HADOOP-12540.03.patch
          JIRA Issue HADOOP-12540
          Optional Tests asflicense javac javadoc mvninstall unit findbugs checkstyle compile
          uname Linux 1d30182d439c 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/patchprocess/apache-yetus-d0f6847/precommit/personality/hadoop.sh
          git revision trunk / 194251c
          Default Java 1.7.0_79
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79
          findbugs v3.0.0
          JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8019/testReport/
          modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
          Max memory used 226MB
          Powered by Apache Yetus http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8019/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s docker + precommit patch detected. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 3m 20s trunk passed +1 compile 0m 15s trunk passed with JDK v1.8.0_66 +1 compile 0m 14s trunk passed with JDK v1.7.0_79 +1 checkstyle 0m 9s trunk passed +1 mvneclipse 0m 16s trunk passed +1 findbugs 0m 33s trunk passed +1 javadoc 0m 13s trunk passed with JDK v1.8.0_66 +1 javadoc 0m 15s trunk passed with JDK v1.7.0_79 +1 mvninstall 0m 17s the patch passed +1 compile 0m 13s the patch passed with JDK v1.8.0_66 +1 javac 0m 13s the patch passed +1 compile 0m 15s the patch passed with JDK v1.7.0_79 +1 javac 0m 15s the patch passed +1 checkstyle 0m 8s the patch passed +1 mvneclipse 0m 13s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 0m 41s the patch passed +1 javadoc 0m 15s the patch passed with JDK v1.8.0_66 +1 javadoc 0m 15s the patch passed with JDK v1.7.0_79 +1 unit 1m 7s hadoop-azure in the patch passed with JDK v1.8.0_66. +1 unit 1m 21s hadoop-azure in the patch passed with JDK v1.7.0_79. +1 asflicense 0m 29s Patch does not generate ASF License warnings. 11m 46s Subsystem Report/Notes Docker Client=1.7.0 Server=1.7.0 Image:test-patch-base-hadoop-date2015-11-04 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12770460/HADOOP-12540.03.patch JIRA Issue HADOOP-12540 Optional Tests asflicense javac javadoc mvninstall unit findbugs checkstyle compile uname Linux 1d30182d439c 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/patchprocess/apache-yetus-d0f6847/precommit/personality/hadoop.sh git revision trunk / 194251c Default Java 1.7.0_79 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79 findbugs v3.0.0 JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8019/testReport/ modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure Max memory used 226MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8019/console This message was automatically generated.
          Hide
          cnauroth Chris Nauroth added a comment -

          Patch v03 looks good. +1 pending Jenkins.

          Show
          cnauroth Chris Nauroth added a comment - Patch v03 looks good. +1 pending Jenkins.
          Hide
          gouravk Gaurav Kanade added a comment -

          Done, yes that was missing, addressed it now. The new v03 patch should have the fix

          Show
          gouravk Gaurav Kanade added a comment - Done, yes that was missing, addressed it now. The new v03 patch should have the fix
          Hide
          cnauroth Chris Nauroth added a comment -

          I see patch v02 now sets leaseID to null, but then there is still an unguarded releaseLease in the finally block. Similar to my last question, is it legal to pass null to the releaseLease call? Intuitively, I would think not, so I was really looking for this in the finally block:

              if (leaseID != null) {
                testAccount.releaseLease(leaseID, fileName);
              }
          

          BTW, I noticed you deleted the old v02 patch and uploaded a new one with the same name. Instead, would you please leave the old files there and use a new file name with the revision number incremented? We prefer to maintain an unaltered history of the patch revisions.

          Thanks again!

          Show
          cnauroth Chris Nauroth added a comment - I see patch v02 now sets leaseID to null , but then there is still an unguarded releaseLease in the finally block. Similar to my last question, is it legal to pass null to the releaseLease call? Intuitively, I would think not, so I was really looking for this in the finally block: if (leaseID != null ) { testAccount.releaseLease(leaseID, fileName); } BTW, I noticed you deleted the old v02 patch and uploaded a new one with the same name. Instead, would you please leave the old files there and use a new file name with the revision number incremented? We prefer to maintain an unaltered history of the patch revisions. Thanks again!
          Hide
          cnauroth Chris Nauroth added a comment -
              String leaseID = "";
          

          If the attempt to create the file or acquire the lease fails with an exception, then leaseID will remain the empty string. Then, in the finally block, it would call testAccount.releaseLease with an empty string. Is that legal? I was expecting to see leaseID set to null before the try block, and then the finally block guards the releaseLease with a check for null before making the call.

          I'll be +1 (pending Jenkins) after that's addressed. Thanks!

          Show
          cnauroth Chris Nauroth added a comment - String leaseID = ""; If the attempt to create the file or acquire the lease fails with an exception, then leaseID will remain the empty string. Then, in the finally block, it would call testAccount.releaseLease with an empty string. Is that legal? I was expecting to see leaseID set to null before the try block, and then the finally block guards the releaseLease with a check for null before making the call. I'll be +1 (pending Jenkins) after that's addressed. Thanks!
          Hide
          gouravk Gaurav Kanade added a comment -

          Thanks Chris Nauroth for the review. Addressed your comments and re-submitting patch now

          Show
          gouravk Gaurav Kanade added a comment - Thanks Chris Nauroth for the review. Addressed your comments and re-submitting patch now
          Hide
          cnauroth Chris Nauroth added a comment -

          Hi Gaurav Kanade. Yes, I think this will work fine overall. A few nitpicks:

          1. The attached patch file appears to be UTF-16. Please make sure to save as ASCII in the next revision. I tested the current patch revision by running iconv -f UTF-16 -t ASCII HADOOP-12540.01.patch.
          2. Please remove the System.out call, which I assume is left over from debugging.
          3. There is a small chance of a resource leak if fs.create succeeds, but then testAccount.acquireShortLease throws an exception. I recommend moving both the fs.create and the testAccount.acquireShortLease within the first try block.

          Thank you!

          Show
          cnauroth Chris Nauroth added a comment - Hi Gaurav Kanade . Yes, I think this will work fine overall. A few nitpicks: The attached patch file appears to be UTF-16. Please make sure to save as ASCII in the next revision. I tested the current patch revision by running iconv -f UTF-16 -t ASCII HADOOP-12540 .01.patch . Please remove the System.out call, which I assume is left over from debugging. There is a small chance of a resource leak if fs.create succeeds, but then testAccount.acquireShortLease throws an exception. I recommend moving both the fs.create and the testAccount.acquireShortLease within the first try block. Thank you!
          Hide
          gouravk Gaurav Kanade added a comment -

          The patch provides a new way to induce client error thus ensuring test functions as expected. Instead of relying on the delete mechanism we simulate the error now by creating a file, then acquiring a short lease on it and then trying to write on it without the lease

          Show
          gouravk Gaurav Kanade added a comment - The patch provides a new way to induce client error thus ensuring test functions as expected. Instead of relying on the delete mechanism we simulate the error now by creating a file, then acquiring a short lease on it and then trying to write on it without the lease
          Hide
          cnauroth Chris Nauroth added a comment -

          This will be a test-only fix. You'll need to come up with a different way to simulate an error so that we can assert the metrics system counted the error correctly.

          Show
          cnauroth Chris Nauroth added a comment - This will be a test-only fix. You'll need to come up with a different way to simulate an error so that we can assert the metrics system counted the error correctly.
          Hide
          cnauroth Chris Nauroth added a comment -

          Gaurav Kanade, I have started seeing failures in TestAzureFileSystemInstrumentation#testClientErrorMetrics since I committed your HADOOP-12508 patch. The failure is intermittent in nature, so I didn't spot the problem before I committed. Would you please fix this test? Note that this is one of the live tests, so you'll need to set up credentials to a live Azure Storage account in azure-test.xml or azure-auth-keys.xml for testing your patch.

          Show
          cnauroth Chris Nauroth added a comment - Gaurav Kanade , I have started seeing failures in TestAzureFileSystemInstrumentation#testClientErrorMetrics since I committed your HADOOP-12508 patch. The failure is intermittent in nature, so I didn't spot the problem before I committed. Would you please fix this test? Note that this is one of the live tests, so you'll need to set up credentials to a live Azure Storage account in azure-test.xml or azure-auth-keys.xml for testing your patch.

            People

            • Assignee:
              gouravk Gaurav Kanade
              Reporter:
              cnauroth Chris Nauroth
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development