[HADOOP-14143] S3A Path Style Being Ignore

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Cannot Reproduce
• Affects Version/s: 2.7.3
• Component/s: fs/s3
• Assignee: Unassigned
• Reporter: Vishnu Vardhan
• Created: 02/Mar/17 23:02
• Updated: 19/Feb/18 17:31
• Resolved: 19/Feb/18 17:31

Description

Hi:

In the following example, path style specification is being ignore

scala>:paste
sc.setLogLevel("DEBUG")
sc.hadoopConfiguration.set("fs.s3a.impl","org.apache.hadoop.fs.s3a.S3AFileSystem")
sc.hadoopConfiguration.set("fs.s3a.endpoint","webscaledemo.netapp.com:8082")
sc.hadoopConfiguration.set("fs.s3a.access.key","")
sc.hadoopConfiguration.set("fs.s3a.secret.key","")
sc.hadoopConfiguration.set("fs.s3a.path.style.access","false")
val s3Rdd = sc.textFile("s3a://myBkt8")
s3Rdd.count()

Debug Log:

application/x-www-form-urlencoded; charset=utf-8
Thu, 02 Mar 2017 22:46:56 GMT
/myBkt8/"
17/03/02 14:46:56 DEBUG request: Sending Request: GET https://webscaledemo.netapp.com:8082 /myBkt8/ Parameters: (max-keys: 1, prefix: user/vardhan/, delimiter: /, ) Headers: (Authorization: AWS 2SNAJYEMQU45YPVYC89D:PIQqLcr6FV61H0+Ay7tw3WygGFo=, User-Agent: aws-sdk-java/1.7.4 Mac_OS_X/10.12.3 Java_HotSpot(TM)_64-Bit_Server_VM/25.60-b23/1.8.0_60, Date: Thu, 02 Mar 2017 22:46:56 GMT, Content-Type: application/x-www-form-urlencoded; charset=utf-8, )
17/03/02 14:46:56 DEBUG PoolingClientConnectionManager: Connection request: [route:

{s}->https://webscaledemo.netapp.com:8082][total kept alive: 0; route allocated: 0 of 15; total allocated: 0 of 15]
17/03/02 14:46:56 DEBUG PoolingClientConnectionManager: Connection leased: [id: 2][route: {s}

->https://webscaledemo.netapp.com:8082][total kept alive: 0; route allocated: 1 of 15; total allocated: 1 of 15]
17/03/02 14:46:56 DEBUG DefaultClientConnectionOperator: Connecting to webscaledemo.netapp.com:8082
17/03/02 14:46:57 DEBUG RequestAddCookies: CookieSpec selected: default
17/03/02 14:46:57 DEBUG RequestAuthCache: Auth cache not set in the context
17/03/02 14:46:57 DEBUG RequestProxyAuthentication: Proxy auth state: UNCHALLENGED
17/03/02 14:46:57 DEBUG SdkHttpClient: Attempt