Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-10511

s3n:// incorrectly handles URLs with secret keys that contain a slash

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: 2.6.0
    • Fix Version/s: None
    • Component/s: fs/s3
    • Labels:

      Description

      This is similar to HADOOP-3733, but happens on s3n:// instead of s3://.

      Essentially if I have a path like "s3n://key:pass%2Fword@example.com/test", it will under certain circumstances be replaced with "s3n://key:pass/test" which then causes "Invalid hostname in URI" exceptions.

      I have a unit test and a fix for this. I'll make a pull request in a moment.

      1. HADOOP-10511.patch
        3 kB
        Daniel Darabos
      2. HADOOP-10511-002.patch
        3 kB
        Steve Loughran

        Issue Links

          Activity

          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user darabos opened a pull request:

          https://github.com/apache/hadoop-common/pull/14

          Fix secret key handling in s3n:// (HADOOP-10511)

          https://issues.apache.org/jira/browse/HADOOP-10511

          I forgot to mention in the commit why I'm adding "example.com" as the default bucket. "s3n:///" is not a valid path, and the URI constructor throws a URISyntaxException for it. I think it is not a great idea anyway to use an invalid path in the tests. I understand the intention of not using something that may be a real bucket, because it could result in accidental damage to production data. But "example.com" is perfect for this, since it cannot exist.

          Let me know if you have other questions about the change. Thanks!

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/darabos/hadoop-common HADOOP-10511

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/hadoop-common/pull/14.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #14


          commit 670926439f7f2abf098fb576f47266564ae50984
          Author: Daniel Darabos <darabos.daniel@gmail.com>
          Date: 2014-04-16T13:00:22Z

          Unit test demonstrating HADOOP-10511. It fails with "expected:<s3n://key:pass/word@example.com/test> but was:<s3n://key:pass/test>".

          commit d7935231d6a9094678dfe9f917ac4733aeaf864b
          Author: Daniel Darabos <darabos.daniel@gmail.com>
          Date: 2014-04-16T13:14:57Z

          Fix HADOOP-10511. URI.create() takes an encoded string. But URI.getAuthority() returns a decoded string. Instead of using URI.create() we need to use the URI constructor which takes a set of URI components.


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user darabos opened a pull request: https://github.com/apache/hadoop-common/pull/14 Fix secret key handling in s3n:// ( HADOOP-10511 ) https://issues.apache.org/jira/browse/HADOOP-10511 I forgot to mention in the commit why I'm adding "example.com" as the default bucket. "s3n:///" is not a valid path, and the URI constructor throws a URISyntaxException for it. I think it is not a great idea anyway to use an invalid path in the tests. I understand the intention of not using something that may be a real bucket, because it could result in accidental damage to production data. But "example.com" is perfect for this, since it cannot exist. Let me know if you have other questions about the change. Thanks! You can merge this pull request into a Git repository by running: $ git pull https://github.com/darabos/hadoop-common HADOOP-10511 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/hadoop-common/pull/14.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #14 commit 670926439f7f2abf098fb576f47266564ae50984 Author: Daniel Darabos <darabos.daniel@gmail.com> Date: 2014-04-16T13:00:22Z Unit test demonstrating HADOOP-10511 . It fails with "expected:<s3n://key:pass/word@example.com/test> but was:<s3n://key:pass/test>". commit d7935231d6a9094678dfe9f917ac4733aeaf864b Author: Daniel Darabos <darabos.daniel@gmail.com> Date: 2014-04-16T13:14:57Z Fix HADOOP-10511 . URI.create() takes an encoded string. But URI.getAuthority() returns a decoded string. Instead of using URI.create() we need to use the URI constructor which takes a set of URI components.
          Hide
          darabos Daniel Darabos added a comment -

          Pull request: https://github.com/apache/hadoop-common/pull/14

          It's strange that this is only pull request 14. I am doing something wrong, right?

          Show
          darabos Daniel Darabos added a comment - Pull request: https://github.com/apache/hadoop-common/pull/14 It's strange that this is only pull request 14. I am doing something wrong, right?
          Hide
          raviprak Ravi Prakash added a comment -

          Daniel! What version of Hadoop are you using? I think in Hadoop 2.2 escaping works

          Show
          raviprak Ravi Prakash added a comment - Daniel! What version of Hadoop are you using? I think in Hadoop 2.2 escaping works
          Hide
          darabos Daniel Darabos added a comment -

          Hi! I'm using 1.0.4 (via Spark 0.9.0). But it shouldn't matter. The unit test demonstrating the problem is against trunk. And even without the unit test, just by looking at the code you can see the bug.

          Show
          darabos Daniel Darabos added a comment - Hi! I'm using 1.0.4 (via Spark 0.9.0). But it shouldn't matter. The unit test demonstrating the problem is against trunk. And even without the unit test, just by looking at the code you can see the bug.
          Hide
          darabos Daniel Darabos added a comment -

          Looks like I'm supposed to attach a patch instead of sending a pull request...?

          Show
          darabos Daniel Darabos added a comment - Looks like I'm supposed to attach a patch instead of sending a pull request...?
          Hide
          darabos Daniel Darabos added a comment -

          Maybe this is how I attach a file.

          Show
          darabos Daniel Darabos added a comment - Maybe this is how I attach a file.
          Hide
          hadoopqa Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12640802/HADOOP-10511.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3812//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3812//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12640802/HADOOP-10511.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 2 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3812//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3812//console This message is automatically generated.
          Hide
          raviprak Ravi Prakash added a comment -

          Hi Daniel!

          Thanks a lot for your contribution. You're right that patch files are how we accept contributions. We usually use SVN style patch file which you can generate from git diff if you use --no-prefix
          https://wiki.apache.org/hadoop/HowToContribute details these protocols.

          Could you please guide me on how I can run the unit test you added? I removed your changes in src/main and ran the unit test using
          $ mvn -Dtest=NativeS3FileSystemContractBaseTest test
          $ mvn -Dtest=NativeS3FileSystemContractBaseTest#testListStatusWithPassword test
          No tests were actually run.

          Show
          raviprak Ravi Prakash added a comment - Hi Daniel! Thanks a lot for your contribution. You're right that patch files are how we accept contributions. We usually use SVN style patch file which you can generate from git diff if you use --no-prefix https://wiki.apache.org/hadoop/HowToContribute details these protocols. Could you please guide me on how I can run the unit test you added? I removed your changes in src/main and ran the unit test using $ mvn -Dtest=NativeS3FileSystemContractBaseTest test $ mvn -Dtest=NativeS3FileSystemContractBaseTest#testListStatusWithPassword test No tests were actually run.
          Hide
          darabos Daniel Darabos added a comment -

          Looks like maybe you need to specify the fully qualified class name:

          $ cd hadoop-common/hadoop-common-project
          $ mvn -Dtest=org.apache.hadoop.fs.s3native.TestInMemoryNativeS3FileSystemContract#testListStatusWithPassword test

          With this I get:

          Failed tests:
          TestInMemoryNativeS3FileSystemContract>NativeS3FileSystemContractBaseTest.testListStatusWithPassword:81 expected:<s3n://key:pass/word@example.com/test> but was:<s3n://key:pass/test>

          Thanks for the pointer to the wiki! I'll upload a --no-prefix patch.

          Show
          darabos Daniel Darabos added a comment - Looks like maybe you need to specify the fully qualified class name: $ cd hadoop-common/hadoop-common-project $ mvn -Dtest=org.apache.hadoop.fs.s3native.TestInMemoryNativeS3FileSystemContract#testListStatusWithPassword test With this I get: Failed tests: TestInMemoryNativeS3FileSystemContract>NativeS3FileSystemContractBaseTest.testListStatusWithPassword:81 expected:<s3n://key:pass/word@example.com/test> but was:<s3n://key:pass/test> Thanks for the pointer to the wiki! I'll upload a --no-prefix patch.
          Hide
          darabos Daniel Darabos added a comment -

          Attached more straightforward patch from "git diff --no-prefix".

          Show
          darabos Daniel Darabos added a comment - Attached more straightforward patch from "git diff --no-prefix".
          Hide
          hadoopqa Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12640895/HADOOP-10511.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3816//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3816//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12640895/HADOOP-10511.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 2 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3816//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3816//console This message is automatically generated.
          Hide
          savu.andrei Andrei Savu added a comment -

          +1 Patch looks good to me.

          Show
          savu.andrei Andrei Savu added a comment - +1 Patch looks good to me.
          Hide
          raviprak Ravi Prakash added a comment -

          Thanks Daniel and Andrei! If this patch touches only S3FileSystem, I'll consider pulling it in. I don't have the necessary background / time / expertise to try to navigate the bigger mess which is HADOOP-3733.

          I'm trying this on trunk after I've applied this patch:

          hadoop fs -ls s3n://<MY_AWS_CRED>:<MY_SECRET_CONTAINING_SLASH>@<MY_BUCKET>/
          -ls: Fatal internal error
          java.lang.NullPointerException
          	at org.apache.hadoop.fs.s3native.NativeS3FileSystem.listStatus(NativeS3FileSystem.java:484)
          	at org.apache.hadoop.fs.shell.PathData.getDirectoryContents(PathData.java:268)
          	at org.apache.hadoop.fs.shell.Command.recursePath(Command.java:351)
          	at org.apache.hadoop.fs.shell.Ls.processPathArgument(Ls.java:98)
          	at org.apache.hadoop.fs.shell.Command.processArgument(Command.java:264)
          	at org.apache.hadoop.fs.shell.Command.processArguments(Command.java:248)
          	at org.apache.hadoop.fs.shell.Command.processRawArguments(Command.java:194)
          	at org.apache.hadoop.fs.shell.Command.run(Command.java:155)
          	at org.apache.hadoop.fs.FsShell.run(FsShell.java:255)
          	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
          	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
          	at org.apache.hadoop.fs.FsShell.main(FsShell.java:308)
          

          The same command where the secret doesn't contain a slash works. Can someone please try it out and confirm / deny this?

          Show
          raviprak Ravi Prakash added a comment - Thanks Daniel and Andrei! If this patch touches only S3FileSystem, I'll consider pulling it in. I don't have the necessary background / time / expertise to try to navigate the bigger mess which is HADOOP-3733 . I'm trying this on trunk after I've applied this patch: hadoop fs -ls s3n: //<MY_AWS_CRED>:<MY_SECRET_CONTAINING_SLASH>@<MY_BUCKET>/ -ls: Fatal internal error java.lang.NullPointerException at org.apache.hadoop.fs.s3native.NativeS3FileSystem.listStatus(NativeS3FileSystem.java:484) at org.apache.hadoop.fs.shell.PathData.getDirectoryContents(PathData.java:268) at org.apache.hadoop.fs.shell.Command.recursePath(Command.java:351) at org.apache.hadoop.fs.shell.Ls.processPathArgument(Ls.java:98) at org.apache.hadoop.fs.shell.Command.processArgument(Command.java:264) at org.apache.hadoop.fs.shell.Command.processArguments(Command.java:248) at org.apache.hadoop.fs.shell.Command.processRawArguments(Command.java:194) at org.apache.hadoop.fs.shell.Command.run(Command.java:155) at org.apache.hadoop.fs.FsShell.run(FsShell.java:255) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) at org.apache.hadoop.fs.FsShell.main(FsShell.java:308) The same command where the secret doesn't contain a slash works. Can someone please try it out and confirm / deny this?
          Hide
          darabos Daniel Darabos added a comment -

          Thanks Ravi! I cannot try it right now, but be sure to replace the slash in the your secret with "%2F".

          Show
          darabos Daniel Darabos added a comment - Thanks Ravi! I cannot try it right now, but be sure to replace the slash in the your secret with "%2F".
          Hide
          raviprak Ravi Prakash added a comment -

          Hi Daniel! I did replace the slash with %2F but still no luck! Please let me know when you get the chance to try it out. Also, do I need to apply any patches other than this one?

          Show
          raviprak Ravi Prakash added a comment - Hi Daniel! I did replace the slash with %2F but still no luck! Please let me know when you get the chance to try it out. Also, do I need to apply any patches other than this one?
          Hide
          darabos Daniel Darabos added a comment -

          Thanks! I get the same exception. It seems that the exception is at least not my fault — I get it both with and without my patch. But it would sure be good to be able to demonstrate the fix from the command line. I'll try to figure out what's wrong.

          Yes, the patch has everything.

          Show
          darabos Daniel Darabos added a comment - Thanks! I get the same exception. It seems that the exception is at least not my fault — I get it both with and without my patch. But it would sure be good to be able to demonstrate the fix from the command line. I'll try to figure out what's wrong. Yes, the patch has everything.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          -1

          about to post the same patch rebased against (the moved) code

          this patch fails against jets3t

          Tests run: 53, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 277.032 sec <<< FAILURE! - in org.apache.hadoop.fs.s3native.Jets3tNativeS3FileSystemContractTest
          testListStatusWithPassword(org.apache.hadoop.fs.s3native.Jets3tNativeS3FileSystemContractTest)  Time elapsed: 0.744 sec  <<< ERROR!
          org.apache.hadoop.security.AccessControlException: Permission denied: s3n://example.com/test
          	at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.processException(Jets3tNativeFileSystemStore.java:449)
          	at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.processException(Jets3tNativeFileSystemStore.java:427)
          	at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.handleException(Jets3tNativeFileSystemStore.java:411)
          	at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.retrieveMetadata(Jets3tNativeFileSystemStore.java:181)
          	at org.apache.hadoop.fs.s3native.NativeS3FileSystem.getFileStatus(NativeS3FileSystem.java:473)
          	at org.apache.hadoop.fs.s3native.NativeS3FileSystem.mkdir(NativeS3FileSystem.java:598)
          	at org.apache.hadoop.fs.s3native.NativeS3FileSystem.mkdirs(NativeS3FileSystem.java:591)
          	at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:1869)
          	at org.apache.hadoop.fs.s3native.NativeS3FileSystemContractBaseTest.testListStatusWithPassword(NativeS3FileSystemContractBaseTest.java:83)
          	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
          	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          	at java.lang.reflect.Method.invoke(Method.java:606)
          	at junit.framework.TestCase.runTest(TestCase.java:176)
          	at junit.framework.TestCase.runBare(TestCase.java:141)
          	at junit.framework.TestResult$1.protect(TestResult.java:122)
          	at junit.framework.TestResult.runProtected(TestResult.java:142)
          	at junit.framework.TestResult.run(TestResult.java:125)
          	at junit.framework.TestCase.run(TestCase.java:129)
          	at junit.framework.TestSuite.runTest(TestSuite.java:255)
          	at junit.framework.TestSuite.run(TestSuite.java:250)
          	at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:84)
          	at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264)
          	at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
          	at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
          	at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200)
          	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
          	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
          Caused by: org.jets3t.service.impl.rest.HttpException: null
          	at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRequest(RestStorageService.java:519)
          	at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRequest(RestStorageService.java:281)
          	at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRestHead(RestStorageService.java:942)
          	at org.jets3t.service.impl.rest.httpclient.RestStorageService.getObjectImpl(RestStorageService.java:2148)
          	at org.jets3t.service.impl.rest.httpclient.RestStorageService.getObjectDetailsImpl(RestStorageService.java:2075)
          	at org.jets3t.service.StorageService.getObjectDetails(StorageService.java:1093)
          	at org.jets3t.service.StorageService.getObjectDetails(StorageService.java:548)
          	at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.retrieveMetadata(Jets3tNativeFileSystemStore.java:174)
          	at org.apache.hadoop.fs.s3native.NativeS3FileSystem.getFileStatus(NativeS3FileSystem.java:473)
          	at org.apache.hadoop.fs.s3native.NativeS3FileSystem.mkdir(NativeS3FileSystem.java:598)
          	at org.apache.hadoop.fs.s3native.NativeS3FileSystem.mkdirs(NativeS3FileSystem.java:591)
          	at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:1869)
          	at org.apache.hadoop.fs.s3native.NativeS3FileSystemContractBaseTest.testListStatusWithPassword(NativeS3FileSystemContractBaseTest.java:83)
          	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
          	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          	at java.lang.reflect.Method.invoke(Method.java:606)
          	at junit.framework.TestCase.runTest(TestCase.java:176)
          	at junit.framework.TestCase.runBare(TestCase.java:141)
          	at junit.framework.TestResult$1.protect(TestResult.java:122)
          	at junit.framework.TestResult.runProtected(TestResult.java:142)
          	at junit.framework.TestResult.run(TestResult.java:125)
          	at junit.framework.TestCase.run(TestCase.java:129)
          	at junit.framework.TestSuite.runTest(TestSuite.java:255)
          	at junit.framework.TestSuite.run(TestSuite.java:250)
          	at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:84)
          	at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264)
          	at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
          	at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
          	at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200)
          	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
          	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
          
          Show
          stevel@apache.org Steve Loughran added a comment - -1 about to post the same patch rebased against (the moved) code this patch fails against jets3t Tests run: 53, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 277.032 sec <<< FAILURE! - in org.apache.hadoop.fs.s3native.Jets3tNativeS3FileSystemContractTest testListStatusWithPassword(org.apache.hadoop.fs.s3native.Jets3tNativeS3FileSystemContractTest) Time elapsed: 0.744 sec <<< ERROR! org.apache.hadoop.security.AccessControlException: Permission denied: s3n: //example.com/test at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.processException(Jets3tNativeFileSystemStore.java:449) at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.processException(Jets3tNativeFileSystemStore.java:427) at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.handleException(Jets3tNativeFileSystemStore.java:411) at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.retrieveMetadata(Jets3tNativeFileSystemStore.java:181) at org.apache.hadoop.fs.s3native.NativeS3FileSystem.getFileStatus(NativeS3FileSystem.java:473) at org.apache.hadoop.fs.s3native.NativeS3FileSystem.mkdir(NativeS3FileSystem.java:598) at org.apache.hadoop.fs.s3native.NativeS3FileSystem.mkdirs(NativeS3FileSystem.java:591) at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:1869) at org.apache.hadoop.fs.s3native.NativeS3FileSystemContractBaseTest.testListStatusWithPassword(NativeS3FileSystemContractBaseTest.java:83) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at junit.framework.TestCase.runTest(TestCase.java:176) at junit.framework.TestCase.runBare(TestCase.java:141) at junit.framework.TestResult$1.protect(TestResult.java:122) at junit.framework.TestResult.runProtected(TestResult.java:142) at junit.framework.TestResult.run(TestResult.java:125) at junit.framework.TestCase.run(TestCase.java:129) at junit.framework.TestSuite.runTest(TestSuite.java:255) at junit.framework.TestSuite.run(TestSuite.java:250) at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:84) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124) at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103) Caused by: org.jets3t.service.impl. rest .HttpException: null at org.jets3t.service.impl. rest .httpclient.RestStorageService.performRequest(RestStorageService.java:519) at org.jets3t.service.impl. rest .httpclient.RestStorageService.performRequest(RestStorageService.java:281) at org.jets3t.service.impl. rest .httpclient.RestStorageService.performRestHead(RestStorageService.java:942) at org.jets3t.service.impl. rest .httpclient.RestStorageService.getObjectImpl(RestStorageService.java:2148) at org.jets3t.service.impl. rest .httpclient.RestStorageService.getObjectDetailsImpl(RestStorageService.java:2075) at org.jets3t.service.StorageService.getObjectDetails(StorageService.java:1093) at org.jets3t.service.StorageService.getObjectDetails(StorageService.java:548) at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.retrieveMetadata(Jets3tNativeFileSystemStore.java:174) at org.apache.hadoop.fs.s3native.NativeS3FileSystem.getFileStatus(NativeS3FileSystem.java:473) at org.apache.hadoop.fs.s3native.NativeS3FileSystem.mkdir(NativeS3FileSystem.java:598) at org.apache.hadoop.fs.s3native.NativeS3FileSystem.mkdirs(NativeS3FileSystem.java:591) at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:1869) at org.apache.hadoop.fs.s3native.NativeS3FileSystemContractBaseTest.testListStatusWithPassword(NativeS3FileSystemContractBaseTest.java:83) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at junit.framework.TestCase.runTest(TestCase.java:176) at junit.framework.TestCase.runBare(TestCase.java:141) at junit.framework.TestResult$1.protect(TestResult.java:122) at junit.framework.TestResult.runProtected(TestResult.java:142) at junit.framework.TestResult.run(TestResult.java:125) at junit.framework.TestCase.run(TestCase.java:129) at junit.framework.TestSuite.runTest(TestSuite.java:255) at junit.framework.TestSuite.run(TestSuite.java:250) at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:84) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124) at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
          Hide
          stevel@apache.org Steve Loughran added a comment -

          patch against trunk, where the s3 code now lives in hadoop-tools/hadoop-aws

          Show
          stevel@apache.org Steve Loughran added a comment - patch against trunk, where the s3 code now lives in hadoop-tools/hadoop-aws
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Patch -002; patch 001 against trunk.

          Jenkins will not catch it as it doesn't run the s3 tests, but this patch will fail the s3 tests. Presumably the new test needs to go down into the TestInMemoryNativeS3FileSystemContract test as in Jets3tNativeS3FileSystemContractTest it gets rejected by AWS

          Show
          stevel@apache.org Steve Loughran added a comment - Patch -002; patch 001 against trunk. Jenkins will not catch it as it doesn't run the s3 tests, but this patch will fail the s3 tests. Presumably the new test needs to go down into the TestInMemoryNativeS3FileSystemContract test as in Jets3tNativeS3FileSystemContractTest it gets rejected by AWS
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12697134/HADOOP-10511-002.patch
          against trunk revision da2fb2b.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The test build failed in hadoop-hdfs-project/hadoop-hdfs

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5618//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5618//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12697134/HADOOP-10511-002.patch against trunk revision da2fb2b. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The test build failed in hadoop-hdfs-project/hadoop-hdfs Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5618//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5618//console This message is automatically generated.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          marking as a duplicate of HADOOP-3733; the fix that went in there is common code, as are the tests and the effort done to try and reduce the printing of secrets.

          Show
          stevel@apache.org Steve Loughran added a comment - marking as a duplicate of HADOOP-3733 ; the fix that went in there is common code, as are the tests and the effort done to try and reduce the printing of secrets.

            People

            • Assignee:
              Unassigned
              Reporter:
              darabos Daniel Darabos
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development