Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-13928

TestAdlFileContextMainOperationsLive.testGetFileContext1 runtime error

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0-alpha2
    • Fix Version/s: 2.9.0, 3.0.0-alpha2, 2.8.2
    • Component/s: fs/adl, test
    • Labels:
      None
    • Environment:

      Mac OS X Sierra 10.12.1

      Description

      {nformat}

      Tests run: 61, Failures: 0, Errors: 1, Skipped: 2, Time elapsed: 102.532 sec <<< FAILURE! - in org.apache.hadoop.fs.adl.live.TestAdlFileContextMainOperationsLive
      testGetFileContext1(org.apache.hadoop.fs.adl.live.TestAdlFileContextMainOperationsLive) Time elapsed: 0.445 sec <<< ERROR!
      java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
      at org.apache.hadoop.fs.AbstractFileSystem.newInstance(AbstractFileSystem.java:136)
      at org.apache.hadoop.fs.AbstractFileSystem.createFileSystem(AbstractFileSystem.java:165)
      at org.apache.hadoop.fs.AbstractFileSystem.get(AbstractFileSystem.java:250)
      at org.apache.hadoop.fs.FileContext$2.run(FileContext.java:331)
      at org.apache.hadoop.fs.FileContext$2.run(FileContext.java:328)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:422)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1857)
      at org.apache.hadoop.fs.FileContext.getAbstractFileSystem(FileContext.java:328)
      at org.apache.hadoop.fs.FileContext.getFSofPath(FileContext.java:320)
      at org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:85)
      at org.apache.hadoop.fs.FileContext.create(FileContext.java:685)
      at org.apache.hadoop.fs.FileContextMainOperationsBaseTest.testGetFileContext1(FileContextMainOperationsBaseTest.java:1350)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:497)
      at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
      at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
      at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
      at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
      at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
      at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
      at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
      at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
      at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
      at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
      at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
      at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
      at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
      at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
      at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
      at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
      at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264)
      at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
      at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
      at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200)
      at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
      at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
      Caused by: java.lang.reflect.InvocationTargetException: null
      at sun.reflect.GeneratedConstructorAccessor14.newInstance(Unknown Source)
      at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
      at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
      at org.apache.hadoop.fs.AbstractFileSystem.newInstance(AbstractFileSystem.java:134)
      at org.apache.hadoop.fs.AbstractFileSystem.createFileSystem(AbstractFileSystem.java:165)
      at org.apache.hadoop.fs.AbstractFileSystem.get(AbstractFileSystem.java:250)
      at org.apache.hadoop.fs.FileContext$2.run(FileContext.java:331)
      at org.apache.hadoop.fs.FileContext$2.run(FileContext.java:328)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:422)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1857)
      at org.apache.hadoop.fs.FileContext.getAbstractFileSystem(FileContext.java:328)
      at org.apache.hadoop.fs.FileContext.getFSofPath(FileContext.java:320)
      at org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:85)
      at org.apache.hadoop.fs.FileContext.create(FileContext.java:685)
      at org.apache.hadoop.fs.FileContextMainOperationsBaseTest.testGetFileContext1(FileContextMainOperationsBaseTest.java:1350)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:497)
      at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
      at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
      at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
      at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
      at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
      at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
      at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
      at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
      at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
      at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
      at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
      at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
      at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
      at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
      at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
      at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
      at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264)
      at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
      at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
      at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200)
      at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
      at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
      Caused by: java.lang.IllegalArgumentException: No value for dfs.adls.oauth2.access.token.provider found in conf file.
      at org.apache.hadoop.fs.adl.AdlFileSystem.getNonEmptyVal(AdlFileSystem.java:935)
      at org.apache.hadoop.fs.adl.AdlFileSystem.getCustomAccessTokenProvider(AdlFileSystem.java:207)
      at org.apache.hadoop.fs.adl.AdlFileSystem.getAccessTokenProvider(AdlFileSystem.java:241)
      at org.apache.hadoop.fs.adl.AdlFileSystem.initialize(AdlFileSystem.java:151)
      at org.apache.hadoop.fs.DelegateToFileSystem.<init>(DelegateToFileSystem.java:52)
      at org.apache.hadoop.fs.adl.Adl.<init>(Adl.java:39)
      at sun.reflect.GeneratedConstructorAccessor14.newInstance(Unknown Source)
      at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
      at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
      at org.apache.hadoop.fs.AbstractFileSystem.newInstance(AbstractFileSystem.java:134)
      at org.apache.hadoop.fs.AbstractFileSystem.createFileSystem(AbstractFileSystem.java:165)
      at org.apache.hadoop.fs.AbstractFileSystem.get(AbstractFileSystem.java:250)
      at org.apache.hadoop.fs.FileContext$2.run(FileContext.java:331)
      at org.apache.hadoop.fs.FileContext$2.run(FileContext.java:328)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:422)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1857)
      at org.apache.hadoop.fs.FileContext.getAbstractFileSystem(FileContext.java:328)
      at org.apache.hadoop.fs.FileContext.getFSofPath(FileContext.java:320)
      at org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:85)
      at org.apache.hadoop.fs.FileContext.create(FileContext.java:685)
      at org.apache.hadoop.fs.FileContextMainOperationsBaseTest.testGetFileContext1(FileContextMainOperationsBaseTest.java:1350)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:497)
      at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
      at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
      at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
      at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
      at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
      at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
      at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
      at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
      at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
      at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
      at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
      at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
      at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
      at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
      at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
      at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
      at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264)
      at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
      at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
      at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200)
      at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
      at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)

      
      

        Issue Links

          Activity

          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          2.8.1 became a security release. Moving fix-version to 2.8.2 after the fact.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - 2.8.1 became a security release. Moving fix-version to 2.8.2 after the fact.
          Hide
          jzhuge John Zhuge added a comment -

          Backport to 2.8 as well.

          Show
          jzhuge John Zhuge added a comment - Backport to 2.8 as well.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11117 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11117/)
          HADOOP-13928. TestAdlFileContextMainOperationsLive.testGetFileContext1 (lei: rev ed09c1418da07a54bb1c5875b31bac47088db56e)

          • (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestDelegateToFsCheckPath.java
          • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/DelegateToFileSystem.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11117 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11117/ ) HADOOP-13928 . TestAdlFileContextMainOperationsLive.testGetFileContext1 (lei: rev ed09c1418da07a54bb1c5875b31bac47088db56e) (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestDelegateToFsCheckPath.java (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/DelegateToFileSystem.java
          Hide
          jzhuge John Zhuge added a comment -

          Thanks Lei (Eddy) Xu for the review and commit!

          Show
          jzhuge John Zhuge added a comment - Thanks Lei (Eddy) Xu for the review and commit!
          Hide
          eddyxu Lei (Eddy) Xu added a comment -

          LGTM, Thanks for the patch, John Zhuge. The fix looks safe to me, and the other failures are not relevant.

          Committed to trunk and branch-2.

          Show
          eddyxu Lei (Eddy) Xu added a comment - LGTM, Thanks for the patch, John Zhuge . The fix looks safe to me, and the other failures are not relevant. Committed to trunk and branch-2.
          Hide
          jzhuge John Zhuge added a comment -

          Akira Ajisaka and Chris Nauroth, could you please review the patch?

          Show
          jzhuge John Zhuge added a comment - Akira Ajisaka and Chris Nauroth , could you please review the patch?
          Hide
          jzhuge John Zhuge added a comment -

          TestViewFsTrash and TestZKDelegationTokenSecretManager pass locally. The unit test failures seem unrelated.

          Show
          jzhuge John Zhuge added a comment - TestViewFsTrash and TestZKDelegationTokenSecretManager pass locally. The unit test failures seem unrelated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 10s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 12m 28s trunk passed
          +1 compile 9m 36s trunk passed
          +1 checkstyle 0m 29s trunk passed
          +1 mvnsite 1m 7s trunk passed
          +1 mvneclipse 0m 17s trunk passed
          +1 findbugs 1m 39s trunk passed
          +1 javadoc 0m 52s trunk passed
          +1 mvninstall 0m 45s the patch passed
          +1 compile 10m 2s the patch passed
          +1 javac 10m 2s the patch passed
          +1 checkstyle 0m 30s the patch passed
          +1 mvnsite 1m 7s the patch passed
          +1 mvneclipse 0m 18s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 45s the patch passed
          +1 javadoc 0m 48s the patch passed
          -1 unit 8m 42s hadoop-common in the patch failed.
          +1 asflicense 0m 32s The patch does not generate ASF License warnings.
          52m 56s



          Reason Tests
          Failed junit tests hadoop.fs.viewfs.TestViewFsTrash
            hadoop.security.token.delegation.TestZKDelegationTokenSecretManager



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HADOOP-13928
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12845284/HADOOP-13928.001.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 78c1297a0319 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 6938b67
          Default Java 1.8.0_111
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11337/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11337/testReport/
          modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11337/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 10s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 12m 28s trunk passed +1 compile 9m 36s trunk passed +1 checkstyle 0m 29s trunk passed +1 mvnsite 1m 7s trunk passed +1 mvneclipse 0m 17s trunk passed +1 findbugs 1m 39s trunk passed +1 javadoc 0m 52s trunk passed +1 mvninstall 0m 45s the patch passed +1 compile 10m 2s the patch passed +1 javac 10m 2s the patch passed +1 checkstyle 0m 30s the patch passed +1 mvnsite 1m 7s the patch passed +1 mvneclipse 0m 18s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 45s the patch passed +1 javadoc 0m 48s the patch passed -1 unit 8m 42s hadoop-common in the patch failed. +1 asflicense 0m 32s The patch does not generate ASF License warnings. 52m 56s Reason Tests Failed junit tests hadoop.fs.viewfs.TestViewFsTrash   hadoop.security.token.delegation.TestZKDelegationTokenSecretManager Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HADOOP-13928 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12845284/HADOOP-13928.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 78c1297a0319 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 6938b67 Default Java 1.8.0_111 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11337/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11337/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11337/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          jzhuge John Zhuge added a comment -

          Patch 001

          • DelegateToFileSystem#getUriDefaultPort returns getDefaultPortIfDefined(fsImpl).
          Show
          jzhuge John Zhuge added a comment - Patch 001 DelegateToFileSystem#getUriDefaultPort returns getDefaultPortIfDefined(fsImpl) .
          Hide
          jzhuge John Zhuge added a comment - - edited

          Did not specify the port in property test.fs.adl.name when I encountered the issue. After adding the default port number 443 to test.fs.adl.name value, the issue went away.

          The issue occurs because AbstractFileSystem#checkPath throws exception for DelegateToFileSystem "adl://CONTAINER.azuredatalakestore.net:443" and path "adl://CONTAINER.azuredatalakestore.net/user/jzhuge/fb37ba52-4e73-43ea-afa5-486097b1c1a4/test".

          A possible fix is to change DelegateToFileSystem#getUriDefaultPort to return getDefaultPortIfDefined(fsImpl).

          Show
          jzhuge John Zhuge added a comment - - edited Did not specify the port in property test.fs.adl.name when I encountered the issue. After adding the default port number 443 to test.fs.adl.name value, the issue went away. The issue occurs because AbstractFileSystem#checkPath throws exception for DelegateToFileSystem "adl://CONTAINER.azuredatalakestore.net:443" and path "adl://CONTAINER.azuredatalakestore.net/user/jzhuge/fb37ba52-4e73-43ea-afa5-486097b1c1a4/test" . A possible fix is to change DelegateToFileSystem#getUriDefaultPort to return getDefaultPortIfDefined(fsImpl) .

            People

            • Assignee:
              jzhuge John Zhuge
              Reporter:
              jzhuge John Zhuge
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development