Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-13513

Java 1.7 support for org.apache.hadoop.fs.azure testcases

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: fs/azure
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Recent improvement on AzureNativeFileSystem rename/delete performance (HADOOP-13403) yielded a test change (HADOOP-13459) which is incompatible with Java 1.7.

      If one tries to include those patches in a Java 1.7 compatible Hadoop tree (e.g. 2.7.x) the following error occurs during test run:

      initializationError(org.apache.hadoop.fs.azure.TestNativeAzureFileSystemClientLogging)  Time elapsed: 0.001 sec  <<< ERROR!
      java.lang.Exception: Class org.apache.hadoop.fs.azure.AbstractWasbTestBase should be public
      	at org.junit.runners.model.FrameworkMethod.validatePublicVoid(FrameworkMethod.java:91)
      	at org.junit.runners.model.FrameworkMethod.validatePublicVoidNoArg(FrameworkMethod.java:70)
      	at org.junit.runners.ParentRunner.validatePublicVoidNoArgMethods(ParentRunner.java:133)
      	at org.junit.runners.BlockJUnit4ClassRunner.validateInstanceMethods(BlockJUnit4ClassRunner.java:165)
      	at org.junit.runners.BlockJUnit4ClassRunner.collectInitializationErrors(BlockJUnit4ClassRunner.java:104)
      	at org.junit.runners.ParentRunner.validate(ParentRunner.java:355)
      	at org.junit.runners.ParentRunner.<init>(ParentRunner.java:76)
      	at org.junit.runners.BlockJUnit4ClassRunner.<init>(BlockJUnit4ClassRunner.java:57)
      	at org.junit.internal.builders.JUnit4Builder.runnerForClass(JUnit4Builder.java:10)
      	at org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:59)
      	at org.junit.internal.builders.AllDefaultPossibilitiesBuilder.runnerForClass(AllDefaultPossibilitiesBuilder.java:26)
      	at org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:59)
      	at org.junit.internal.requests.ClassRequest.getRunner(ClassRequest.java:26)
      	at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:262)
      	at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
      	at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
      	at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200)
      	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
      	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
      

      The problem can be resolved by setting AbstractWasbTestBase to public.

        Issue Links

          Activity

          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 12s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 6m 38s trunk passed
          +1 compile 0m 17s trunk passed
          +1 checkstyle 0m 13s trunk passed
          +1 mvnsite 0m 20s trunk passed
          +1 mvneclipse 0m 21s trunk passed
          +1 findbugs 0m 25s trunk passed
          +1 javadoc 0m 13s trunk passed
          +1 mvninstall 0m 16s the patch passed
          +1 compile 0m 14s the patch passed
          +1 javac 0m 14s the patch passed
          +1 checkstyle 0m 10s the patch passed
          +1 mvnsite 0m 18s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 0m 30s the patch passed
          +1 javadoc 0m 10s the patch passed
          +1 unit 1m 18s hadoop-azure in the patch passed.
          +1 asflicense 0m 15s The patch does not generate ASF License warnings.
          13m 16s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12824303/HADOOP-13513-001.patch
          JIRA Issue HADOOP-13513
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 6b97ebad5190 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 913a895
          Default Java 1.8.0_101
          findbugs v3.0.0
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10296/testReport/
          modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10296/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 12s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 38s trunk passed +1 compile 0m 17s trunk passed +1 checkstyle 0m 13s trunk passed +1 mvnsite 0m 20s trunk passed +1 mvneclipse 0m 21s trunk passed +1 findbugs 0m 25s trunk passed +1 javadoc 0m 13s trunk passed +1 mvninstall 0m 16s the patch passed +1 compile 0m 14s the patch passed +1 javac 0m 14s the patch passed +1 checkstyle 0m 10s the patch passed +1 mvnsite 0m 18s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 30s the patch passed +1 javadoc 0m 10s the patch passed +1 unit 1m 18s hadoop-azure in the patch passed. +1 asflicense 0m 15s The patch does not generate ASF License warnings. 13m 16s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12824303/HADOOP-13513-001.patch JIRA Issue HADOOP-13513 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 6b97ebad5190 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 913a895 Default Java 1.8.0_101 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10296/testReport/ modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10296/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Tibor, looks simple, but even for a one-liner we need to follow the object store patch policy: which Azure endpoint did you test against?

          Show
          stevel@apache.org Steve Loughran added a comment - Tibor, looks simple, but even for a one-liner we need to follow the object store patch policy: which Azure endpoint did you test against?
          Hide
          tibor.kiss@gmail.com Tibor Kiss added a comment -

          Thanks Steve Loughran for the quick review.
          I was not aware that we need to do Azure endpoint testing. I did not find any details on how to test it manually. Could you please give me some pointers to Azure related testing?

          Show
          tibor.kiss@gmail.com Tibor Kiss added a comment - Thanks Steve Loughran for the quick review. I was not aware that we need to do Azure endpoint testing. I did not find any details on how to test it manually. Could you please give me some pointers to Azure related testing?
          Hide
          cnauroth Chris Nauroth added a comment -

          Hello Tibor Kiss. Steve is referring to this policy in the Hadoop contribution wiki:

          https://wiki.apache.org/hadoop/HowToContribute#Submitting_patches_against_object_stores_such_as_Amazon_S3.2C_OpenStack_Swift_and_Microsoft_Azure

          To summarize, for modules like hadoop-azure, hadoop-aws, etc., there are either no unit tests running during Apache pre-commit, or there are unit tests in the module, but we consider them insufficient to validate a patch for commit. Instead, we require the contributor to run integration tests directly against the back-end service. These are still structured as JUnit tests under the src/test/java directory, but the tests get skipped unless the developer does some extra configuration to make the necessary credentials available. The exact configuration process varies per file system:

          http://hadoop.apache.org/docs/r2.7.2/hadoop-aws/tools/hadoop-aws/index.html

          http://hadoop.apache.org/docs/r2.7.2/hadoop-azure/index.html#Testing_the_hadoop-azure_Module

          For this patch, I'm going to kick off my own test run for you now. We currently have a test failure on branch-2 and branch-2.8 because of this, so I want to get it fixed quickly. You can keep this information in mind for future patches though.

          Show
          cnauroth Chris Nauroth added a comment - Hello Tibor Kiss . Steve is referring to this policy in the Hadoop contribution wiki: https://wiki.apache.org/hadoop/HowToContribute#Submitting_patches_against_object_stores_such_as_Amazon_S3.2C_OpenStack_Swift_and_Microsoft_Azure To summarize, for modules like hadoop-azure, hadoop-aws, etc., there are either no unit tests running during Apache pre-commit, or there are unit tests in the module, but we consider them insufficient to validate a patch for commit. Instead, we require the contributor to run integration tests directly against the back-end service. These are still structured as JUnit tests under the src/test/java directory, but the tests get skipped unless the developer does some extra configuration to make the necessary credentials available. The exact configuration process varies per file system: http://hadoop.apache.org/docs/r2.7.2/hadoop-aws/tools/hadoop-aws/index.html http://hadoop.apache.org/docs/r2.7.2/hadoop-azure/index.html#Testing_the_hadoop-azure_Module For this patch, I'm going to kick off my own test run for you now. We currently have a test failure on branch-2 and branch-2.8 because of this, so I want to get it fixed quickly. You can keep this information in mind for future patches though.
          Hide
          cnauroth Chris Nauroth added a comment -

          +1 for the patch. I ran all tests successfully against an Azure Storage account in West US, using both JDK 8 and JDK 7. I have committed this to trunk, branch-2 and branch-2.8. Tibor Kiss, thank you for fixing a bug I introduced. Steve Loughran, thank you for help with the code review.

          Show
          cnauroth Chris Nauroth added a comment - +1 for the patch. I ran all tests successfully against an Azure Storage account in West US, using both JDK 8 and JDK 7. I have committed this to trunk, branch-2 and branch-2.8. Tibor Kiss , thank you for fixing a bug I introduced. Steve Loughran , thank you for help with the code review.
          Hide
          tibor.kiss@gmail.com Tibor Kiss added a comment -

          Thanks Chris Nauroth and Steve Loughran for the prompt review!

          Show
          tibor.kiss@gmail.com Tibor Kiss added a comment - Thanks Chris Nauroth and Steve Loughran for the prompt review!
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10301 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10301/)
          HADOOP-13513. Java 1.7 support for org.apache.hadoop.fs.azure testcases. (cnauroth: rev ae4db2544346370404826d5b55b2678f5f92fe1f)

          • (edit) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/AbstractWasbTestBase.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10301 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10301/ ) HADOOP-13513 . Java 1.7 support for org.apache.hadoop.fs.azure testcases. (cnauroth: rev ae4db2544346370404826d5b55b2678f5f92fe1f) (edit) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/AbstractWasbTestBase.java

            People

            • Assignee:
              tibor.kiss@gmail.com Tibor Kiss
              Reporter:
              tibor.kiss@gmail.com Tibor Kiss
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development