Brahma Reddy Battula added a comment - 22/Feb/15 17:30 Akira Ajisaka Kindly check attached patch for removal of http client from open-stack project..

Hadoop QA added a comment - 22/Feb/15 18:13 -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12700103/HADOOP-11614.patch against trunk revision fe7a302. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 3 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to introduce 1 new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-tools/hadoop-openstack. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5756//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/5756//artifact/patchprocess/newPatchFindbugsWarningshadoop-openstack.html Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5756//console This message is automatically generated.

Akira Ajisaka added a comment - 24/Feb/15 23:09 Hi Brahma Reddy Battula , thank you for the patch. Now I have two concerns with the big change. 1. The patch changes the signature of some public methods. That is an incompatible change. 2. OpenStack community maintains another SwiftNativeFileSystem in https://github.com/openstack/sahara-extra . IMHO, I don't want code duplication and want to merge them first if possible. I asked Kazuki OIKAWA , who maintains the project, to tell me what we should to do. He thinks it would be better to drop hadoop-openstack codebase and use the code in OpenStack community since these developments have not been in-sync. I mostly agree with him if the below conditions are satisfied. sahara-extra releases its jar periodically and we can use it from Maven repository. Create a minimum wrapper code and call sahara-extra APIs from it, like s3n or azure. Keep the contract tests for hadoop-openstack. Hi Steve Loughran , what do you think?

Steve Loughran added a comment - 24/Feb/15 23:52 I think that openstack maintenance is something we should discuss with them before these changes. I never knew about their work, it hasn't picked up the Hadoop 2.5+ changes (contracts and fixes), but may have other changes of its own —it looks like later auth changes are in there. So a merge is the right thing to do, as long as their versions are compatible with the hadoop releases (requirement on us as well as them)

Brahma Reddy Battula added a comment - 28/Feb/15 12:09 Thanks a lot Akira Ajisaka and Steve Loughran for comments.. Yes, this is big change and there are incompatible changes. sahara-extra releases its jar periodically and we can use it from Maven repository. Create a minimum wrapper code and call sahara-extra APIs from it, like s3n or azure. Keep the contract tests for hadoop-openstack. this sounds good to me.As Open stack people also agreed. So a merge is the right thing to do, as long as their versions are compatible with the hadoop releases (requirement on us as well as them) This also ok but need to maintain compatible both the ends..

Wei-Chiu Chuang added a comment - 14/Jan/16 18:28 Hi Brahma Reddy Battula and Akira AJISAKA thanks for working on this. Are there any updates on this work? I'd like to join the effort and push this change into codebase.

Hadoop QA added a comment - 14/Jan/16 19:36 -1 overall Vote Subsystem Runtime Comment -1 patch 0m 5s HADOOP-11614 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12700103/HADOOP-11614.patch JIRA Issue HADOOP-11614 Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8408/console This message was automatically generated.

Akira Ajisaka added a comment - 16/Jan/16 19:24 Are there any updates on this work? I'd like to join the effort and push this change into codebase. Brahma Reddy Battula , how is this issue going on? I'm not working on this issue now.

Wei-Chiu Chuang added a comment - 04/May/16 18:42 Pinging again – should we file a new jira to create wrappers around sahara-extra before moving on with this? Thanks

Hadoop QA added a comment - 04/May/16 19:26 -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 patch 0m 5s HADOOP-11614 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12700103/HADOOP-11614.patch JIRA Issue HADOOP-11614 Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9277/console Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.

Steve Loughran added a comment - 19/May/16 13:02 Patch 002; patch 001 + fixed to compile against trunk remove all references to httpclient from the openstack pom inserted HADOOP-12767 to have build/test runs against the target versions. fixed up some of the inevitable javadoc warnings that checkstyle will have kittens about. This patch fails to run, and fails differently against separate infras. As such: -1

Steve Loughran added a comment - 19/May/16 13:05 All tests are failing during auth, which is inevitably the first HTTP(S) conversation with the Openstack servers. Against Rackspace UK T E S T S ------------------------------------------------------- Running org.apache.hadoop.fs.swift.http.TestSwiftRestClient Tests run: 3, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 2.513 sec <<< FAILURE! - in org.apache.hadoop.fs.swift.http.TestSwiftRestClient testAuthenticate(org.apache.hadoop.fs.swift.http.TestSwiftRestClient) Time elapsed: 1.891 sec <<< ERROR! org.apache.hadoop.fs.swift.exceptions.SwiftInvalidResponseException: Method POST on https: //lon.identity.api.rackspacecloud.com/v2.0/ failed, status code: 405, status line: HTTP/1.1 405 Method Not Allowed at org.apache.hadoop.fs.swift.http.SwiftRestClient.buildException(SwiftRestClient.java:1513) at org.apache.hadoop.fs.swift.http.SwiftRestClient.perform(SwiftRestClient.java:1414) at org.apache.hadoop.fs.swift.http.SwiftRestClient.authenticate(SwiftRestClient.java:1090) at org.apache.hadoop.fs.swift.http.TestSwiftRestClient.testAuthenticate(TestSwiftRestClient.java:77) testPutAndDelete(org.apache.hadoop.fs.swift.http.TestSwiftRestClient) Time elapsed: 0.383 sec <<< ERROR! org.apache.hadoop.fs.swift.exceptions.SwiftInvalidResponseException: Method POST on https: //lon.identity.api.rackspacecloud.com/v2.0/ failed, status code: 405, status line: HTTP/1.1 405 Method Not Allowed at org.apache.hadoop.fs.swift.http.SwiftRestClient.buildException(SwiftRestClient.java:1513) at org.apache.hadoop.fs.swift.http.SwiftRestClient.perform(SwiftRestClient.java:1414) at org.apache.hadoop.fs.swift.http.SwiftRestClient.authenticate(SwiftRestClient.java:1090) at org.apache.hadoop.fs.swift.http.TestSwiftRestClient.testPutAndDelete(TestSwiftRestClient.java:84) Results : Tests in error: TestSwiftRestClient.testAuthenticate:77 » SwiftInvalidResponse Method POST IBM softlayer ------------------------------------------------------- Running org.apache.hadoop.fs.swift.http.TestSwiftRestClient Tests run: 3, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 2.848 sec <<< FAILURE! - in org.apache.hadoop.fs.swift.http.TestSwiftRestClient testAuthenticate(org.apache.hadoop.fs.swift.http.TestSwiftRestClient) Time elapsed: 2.026 sec <<< ERROR! java.lang.IllegalStateException: Invalid use of BasicClientConnManager: connection still allocated. Make sure to release the connection before allocating another one. at org.apache.http.util.Asserts.check(Asserts.java:34) at org.apache.http.impl.conn.BasicClientConnectionManager.getConnection(BasicClientConnectionManager.java:166) at org.apache.http.impl.conn.BasicClientConnectionManager$1.getConnection(BasicClientConnectionManager.java:148) at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:424) at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:884) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55) at org.apache.hadoop.fs.swift.http.SwiftRestClient.execWithDebugOutput(SwiftRestClient.java:1723) at org.apache.hadoop.fs.swift.http.SwiftRestClient.exec(SwiftRestClient.java:1678) at org.apache.hadoop.fs.swift.http.SwiftRestClient.perform(SwiftRestClient.java:1397) at org.apache.hadoop.fs.swift.http.SwiftRestClient.authenticate(SwiftRestClient.java:1090) at org.apache.hadoop.fs.swift.http.TestSwiftRestClient.testAuthenticate(TestSwiftRestClient.java:77) testPutAndDelete(org.apache.hadoop.fs.swift.http.TestSwiftRestClient) Time elapsed: 0.608 sec <<< ERROR! java.lang.IllegalStateException: Invalid use of BasicClientConnManager: connection still allocated. Make sure to release the connection before allocating another one. at org.apache.http.util.Asserts.check(Asserts.java:34) at org.apache.http.impl.conn.BasicClientConnectionManager.getConnection(BasicClientConnectionManager.java:166) at org.apache.http.impl.conn.BasicClientConnectionManager$1.getConnection(BasicClientConnectionManager.java:148) at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:424) at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:884) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55) at org.apache.hadoop.fs.swift.http.SwiftRestClient.execWithDebugOutput(SwiftRestClient.java:1723) at org.apache.hadoop.fs.swift.http.SwiftRestClient.exec(SwiftRestClient.java:1678) at org.apache.hadoop.fs.swift.http.SwiftRestClient.perform(SwiftRestClient.java:1397) at org.apache.hadoop.fs.swift.http.SwiftRestClient.authenticate(SwiftRestClient.java:1090) at org.apache.hadoop.fs.swift.http.TestSwiftRestClient.testPutAndDelete(TestSwiftRestClient.java:84) Against Rackspace US: infinite loop. " Thread -1" prio=5 tid=0x00007fcf191f1800 nid=0x5103 runnable [0x00007000013c9000] java.lang. Thread .State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:152) at java.net.SocketInputStream.read(SocketInputStream.java:122) at sun.security.ssl.InputRecord.readFully(InputRecord.java:442) at sun.security.ssl.InputRecord.read(InputRecord.java:480) at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:934) - locked <0x00000007d8073328> (a java.lang. Object ) at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:891) at sun.security.ssl.AppInputStream.read(AppInputStream.java:102) - locked <0x00000007d8098cf8> (a sun.security.ssl.AppInputStream) at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160) at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84) at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283) at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:259) at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:209) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:686) at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:488) at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:884) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55) at org.apache.hadoop.fs.swift.http.SwiftRestClient.execWithDebugOutput(SwiftRestClient.java:1723) at org.apache.hadoop.fs.swift.http.SwiftRestClient.exec(SwiftRestClient.java:1662) at org.apache.hadoop.fs.swift.http.SwiftRestClient.perform(SwiftRestClient.java:1397) at org.apache.hadoop.fs.swift.http.SwiftRestClient.authenticate(SwiftRestClient.java:1090) at org.apache.hadoop.fs.swift.http.SwiftRestClient.exec(SwiftRestClient.java:1696) at org.apache.hadoop.fs.swift.http.SwiftRestClient.perform(SwiftRestClient.java:1397) at org.apache.hadoop.fs.swift.http.SwiftRestClient.headRequest(SwiftRestClient.java:1026) at org.apache.hadoop.fs.swift.http.SwiftRestClient.createContainer(SwiftRestClient.java:1264) at org.apache.hadoop.fs.swift.http.SwiftRestClient.createDefaultContainer(SwiftRestClient.java:1249) - locked <0x00000007d9580e68> (a org.apache.hadoop.fs.swift.http.SwiftRestClient) at org.apache.hadoop.fs.swift.http.SwiftRestClient.access$1500(SwiftRestClient.java:98) at org.apache.hadoop.fs.swift.http.SwiftRestClient$AuthenticationPost.extractResult(SwiftRestClient.java:1226) at org.apache.hadoop.fs.swift.http.SwiftRestClient$AuthenticationPost.extractResult(SwiftRestClient.java:1095) at org.apache.hadoop.fs.swift.http.SwiftRestClient.perform(SwiftRestClient.java:1418) at org.apache.hadoop.fs.swift.http.SwiftRestClient.authenticate(SwiftRestClient.java:1090) at org.apache.hadoop.fs.swift.http.SwiftRestClient.exec(SwiftRestClient.java:1696) at org.apache.hadoop.fs.swift.http.SwiftRestClient.perform(SwiftRestClient.java:1397) at org.apache.hadoop.fs.swift.http.SwiftRestClient.headRequest(SwiftRestClient.java:1026) at org.apache.hadoop.fs.swift.http.SwiftRestClient.createContainer(SwiftRestClient.java:1264) at org.apache.hadoop.fs.swift.http.SwiftRestClient.createDefaultContainer(SwiftRestClient.java:1249) - locked <0x00000007d9580e68> (a org.apache.hadoop.fs.swift.http.SwiftRestClient) at org.apache.hadoop.fs.swift.http.SwiftRestClient.access$1500(SwiftRestClient.java:98) at org.apache.hadoop.fs.swift.http.SwiftRestClient$AuthenticationPost.extractResult(SwiftRestClient.java:1226) at org.apache.hadoop.fs.swift.http.SwiftRestClient$AuthenticationPost.extractResult(SwiftRestClient.java:1095) at org.apache.hadoop.fs.swift.http.SwiftRestClient.perform(SwiftRestClient.java:1418) at org.apache.hadoop.fs.swift.http.SwiftRestClient.authenticate(SwiftRestClient.java:1090) at org.apache.hadoop.fs.swift.http.SwiftRestClient.exec(SwiftRestClient.java:1696) at org.apache.hadoop.fs.swift.http.SwiftRestClient.perform(SwiftRestClient.java:1397) at org.apache.hadoop.fs.swift.http.SwiftRestClient.headRequest(SwiftRestClient.java:1026) at org.apache.hadoop.fs.swift.http.SwiftRestClient.createContainer(SwiftRestClient.java:1264) at org.apache.hadoop.fs.swift.http.SwiftRestClient.createDefaultContainer(SwiftRestClient.java:1249) - locked <0x00000007d9580e68> (a org.apache.hadoop.fs.swift.http.SwiftRestClient) at org.apache.hadoop.fs.swift.http.SwiftRestClient.access$1500(SwiftRestClient.java:98) at org.apache.hadoop.fs.swift.http.SwiftRestClient$AuthenticationPost.extractResult(SwiftRestClient.java:1226) at org.apache.hadoop.fs.swift.http.SwiftRestClient$AuthenticationPost.extractResult(SwiftRestClient.java:1095) at org.apache.hadoop.fs.swift.http.SwiftRestClient.perform(SwiftRestClient.java:1418) at org.apache.hadoop.fs.swift.http.SwiftRestClient.authenticate(SwiftRestClient.java:1090) at org.apache.hadoop.fs.swift.http.SwiftRestClient.exec(SwiftRestClient.java:1696) at org.apache.hadoop.fs.swift.http.SwiftRestClient.perform(SwiftRestClient.java:1397) at org.apache.hadoop.fs.swift.http.SwiftRestClient.headRequest(SwiftRestClient.java:1026) at org.apache.hadoop.fs.swift.http.SwiftRestClient.createContainer(SwiftRestClient.java:1264) at org.apache.hadoop.fs.swift.http.SwiftRestClient.createDefaultContainer(SwiftRestClient.java:1249) - locked <0x00000007d9580e68> (a org.apache.hadoop.fs.swift.http.SwiftRestClient) at org.apache.hadoop.fs.swift.http.SwiftRestClient.access$1500(SwiftRestClient.java:98) at org.apache.hadoop.fs.swift.http.SwiftRestClient$AuthenticationPost.extractResult(SwiftRestClient.java:1226) at org.apache.hadoop.fs.swift.http.SwiftRestClient$AuthenticationPost.extractResult(SwiftRestClient.java:1095) at org.apache.hadoop.fs.swift.http.SwiftRestClient.perform(SwiftRestClient.java:1418) at org.apache.hadoop.fs.swift.http.SwiftRestClient.authenticate(SwiftRestClient.java:1090) at org.apache.hadoop.fs.swift.http.SwiftRestClient.exec(SwiftRestClient.java:1696) at org.apache.hadoop.fs.swift.http.SwiftRestClient.perform(SwiftRestClient.java:1397) at org.apache.hadoop.fs.swift.http.SwiftRestClient.headRequest(SwiftRestClient.java:1026) at org.apache.hadoop.fs.swift.http.SwiftRestClient.createContainer(SwiftRestClient.java:1264) at org.apache.hadoop.fs.swift.http.SwiftRestClient.createDefaultContainer(SwiftRestClient.java:1249) - locked <0x00000007d9580e68> (a org.apache.hadoop.fs.swift.http.SwiftRestClient) at org.apache.hadoop.fs.swift.http.SwiftRestClient.access$1500(SwiftRestClient.java:98) at org.apache.hadoop.fs.swift.http.SwiftRestClient$AuthenticationPost.extractResult(SwiftRestClient.java:1226) at org.apache.hadoop.fs.swift.http.SwiftRestClient$AuthenticationPost.extractResult(SwiftRestClient.java:1095) at org.apache.hadoop.fs.swift.http.SwiftRestClient.perform(SwiftRestClient.java:1418) at org.apache.hadoop.fs.swift.http.SwiftRestClient.authenticate(SwiftRestClient.java:1090) at org.apache.hadoop.fs.swift.http.SwiftRestClient.exec(SwiftRestClient.java:1696) at org.apache.hadoop.fs.swift.http.SwiftRestClient.perform(SwiftRestClient.java:1397) at org.apache.hadoop.fs.swift.http.SwiftRestClient.headRequest(SwiftRestClient.java:1026) at org.apache.hadoop.fs.swift.http.SwiftRestClient.createContainer(SwiftRestClient.java:1264) at org.apache.hadoop.fs.swift.http.SwiftRestClient.createDefaultContainer(SwiftRestClient.java:1249) - locked <0x00000007d9580e68> (a org.apache.hadoop.fs.swift.http.SwiftRestClient) at org.apache.hadoop.fs.swift.http.SwiftRestClient.access$1500(SwiftRestClient.java:98) at org.apache.hadoop.fs.swift.http.SwiftRestClient$AuthenticationPost.extractResult(SwiftRestClient.java:1226) at org.apache.hadoop.fs.swift.http.SwiftRestClient$AuthenticationPost.extractResult(SwiftRestClient.java:1095) at org.apache.hadoop.fs.swift.http.SwiftRestClient.perform(SwiftRestClient.java:1418) at org.apache.hadoop.fs.swift.http.SwiftRestClient.authenticate(SwiftRestClient.java:1090) at org.apache.hadoop.fs.swift.http.SwiftRestClient.exec(SwiftRestClient.java:1696) at org.apache.hadoop.fs.swift.http.SwiftRestClient.perform(SwiftRestClient.java:1397) at org.apache.hadoop.fs.swift.http.SwiftRestClient.headRequest(SwiftRestClient.java:1026) at org.apache.hadoop.fs.swift.http.SwiftRestClient.createContainer(SwiftRestClient.java:1264) at org.apache.hadoop.fs.swift.http.SwiftRestClient.createDefaultContainer(SwiftRestClient.java:1249) - locked <0x00000007d9580e68> (a org.apache.hadoop.fs.swift.http.SwiftRestClient) at org.apache.hadoop.fs.swift.http.SwiftRestClient.access$1500(SwiftRestClient.java:98) at org.apache.hadoop.fs.swift.http.SwiftRestClient$AuthenticationPost.extractResult(SwiftRestClient.java:1226) at org.apache.hadoop.fs.swift.http.SwiftRestClient$AuthenticationPost.extractResult(SwiftRestClient.java:1095) at org.apache.hadoop.fs.swift.http.SwiftRestClient.perform(SwiftRestClient.java:1418) at org.apache.hadoop.fs.swift.http.SwiftRestClient.authenticate(SwiftRestClient.java:1090) at org.apache.hadoop.fs.swift.http.SwiftRestClient.exec(SwiftRestClient.java:1696) at org.apache.hadoop.fs.swift.http.SwiftRestClient.perform(SwiftRestClient.java:1397) at org.apache.hadoop.fs.swift.http.SwiftRestClient.headRequest(SwiftRestClient.java:1026) at org.apache.hadoop.fs.swift.http.SwiftRestClient.createContainer(SwiftRestClient.java:1264) at org.apache.hadoop.fs.swift.http.SwiftRestClient.createDefaultContainer(SwiftRestClient.java:1249) - locked <0x00000007d9580e68> (a org.apache.hadoop.fs.swift.http.SwiftRestClient) at org.apache.hadoop.fs.swift.http.SwiftRestClient.access$1500(SwiftRestClient.java:98) at org.apache.hadoop.fs.swift.http.SwiftRestClient$AuthenticationPost.extractResult(SwiftRestClient.java:1226) at org.apache.hadoop.fs.swift.http.SwiftRestClient$AuthenticationPost.extractResult(SwiftRestClient.java:1095) at org.apache.hadoop.fs.swift.http.SwiftRestClient.perform(SwiftRestClient.java:1418) at org.apache.hadoop.fs.swift.http.SwiftRestClient.authenticate(SwiftRestClient.java:1090) at org.apache.hadoop.fs.swift.http.SwiftRestClient.exec(SwiftRestClient.java:1696) at org.apache.hadoop.fs.swift.http.SwiftRestClient.perform(SwiftRestClient.java:1397) at org.apache.hadoop.fs.swift.http.SwiftRestClient.headRequest(SwiftRestClient.java:1026) at org.apache.hadoop.fs.swift.http.SwiftRestClient.createContainer(SwiftRestClient.java:1264) at org.apache.hadoop.fs.swift.http.SwiftRestClient.createDefaultContainer(SwiftRestClient.java:1249) - locked <0x00000007d9580e68> (a org.apache.hadoop.fs.swift.http.SwiftRestClient) at org.apache.hadoop.fs.swift.http.SwiftRestClient.access$1500(SwiftRestClient.java:98) at org.apache.hadoop.fs.swift.http.SwiftRestClient$AuthenticationPost.extractResult(SwiftRestClient.java:1226) at org.apache.hadoop.fs.swift.http.SwiftRestClient$AuthenticationPost.extractResult(SwiftRestClient.java:1095) at org.apache.hadoop.fs.swift.http.SwiftRestClient.perform(SwiftRestClient.java:1418) at org.apache.hadoop.fs.swift.http.SwiftRestClient.authenticate(SwiftRestClient.java:1090) at org.apache.hadoop.fs.swift.http.SwiftRestClient.exec(SwiftRestClient.java:1696) at org.apache.hadoop.fs.swift.http.SwiftRestClient.perform(SwiftRestClient.java:1397) at org.apache.hadoop.fs.swift.http.SwiftRestClient.headRequest(SwiftRestClient.java:1026) at org.apache.hadoop.fs.swift.http.SwiftRestClient.createContainer(SwiftRestClient.java:1264) at org.apache.hadoop.fs.swift.http.SwiftRestClient.createDefaultContainer(SwiftRestClient.java:1249) - locked <0x00000007d9580e68> (a org.apache.hadoop.fs.swift.http.SwiftRestClient) at org.apache.hadoop.fs.swift.http.SwiftRestClient.access$1500(SwiftRestClient.java:98) at org.apache.hadoop.fs.swift.http.SwiftRestClient$AuthenticationPost.extractResult(SwiftRestClient.java:1226) at org.apache.hadoop.fs.swift.http.SwiftRestClient$AuthenticationPost.extractResult(SwiftRestClient.java:1095) at org.apache.hadoop.fs.swift.http.SwiftRestClient.perform(SwiftRestClient.java:1418) at org.apache.hadoop.fs.swift.http.SwiftRestClient.authenticate(SwiftRestClient.java:1090) at org.apache.hadoop.fs.swift.http.SwiftRestClient.exec(SwiftRestClient.java:1696) at org.apache.hadoop.fs.swift.http.SwiftRestClient.perform(SwiftRestClient.java:1397) at org.apache.hadoop.fs.swift.http.SwiftRestClient.headRequest(SwiftRestClient.java:1026) at org.apache.hadoop.fs.swift.http.SwiftRestClient.createContainer(SwiftRestClient.java:1264) at org.apache.hadoop.fs.swift.http.SwiftRestClient.createDefaultContainer(SwiftRestClient.java:1249) - locked <0x00000007d9580e68> (a org.apache.hadoop.fs.swift.http.SwiftRestClient) at org.apache.hadoop.fs.swift.http.SwiftRestClient.access$1500(SwiftRestClient.java:98) at org.apache.hadoop.fs.swift.http.SwiftRestClient$AuthenticationPost.extractResult(SwiftRestClient.java:1226) at org.apache.hadoop.fs.swift.http.SwiftRestClient$AuthenticationPost.extractResult(SwiftRestClient.java:1095) at org.apache.hadoop.fs.swift.http.SwiftRestClient.perform(SwiftRestClient.java:1418) at org.apache.hadoop.fs.swift.http.SwiftRestClient.authenticate(SwiftRestClient.java:1090) at org.apache.hadoop.fs.swift.http.SwiftRestClient.exec(SwiftRestClient.java:1696) at org.apache.hadoop.fs.swift.http.SwiftRestClient.perform(SwiftRestClient.java:1397) at org.apache.hadoop.fs.swift.http.SwiftRestClient.headRequest(SwiftRestClient.java:1026) at org.apache.hadoop.fs.swift.http.SwiftRestClient.createContainer(SwiftRestClient.java:1264) at org.apache.hadoop.fs.swift.http.SwiftRestClient.createDefaultContainer(SwiftRestClient.java:1249) - locked <0x00000007d9580e68> (a org.apache.hadoop.fs.swift.http.SwiftRestClient) at org.apache.hadoop.fs.swift.http.SwiftRestClient.access$1500(SwiftRestClient.java:98) at org.apache.hadoop.fs.swift.http.SwiftRestClient$AuthenticationPost.extractResult(SwiftRestClient.java:1226) at org.apache.hadoop.fs.swift.http.SwiftRestClient$AuthenticationPost.extractResult(SwiftRestClient.java:1095) at org.apache.hadoop.fs.swift.http.SwiftRestClient.perform(SwiftRestClient.java:1418) at org.apache.hadoop.fs.swift.http.SwiftRestClient.authenticate(SwiftRestClient.java:1090) at org.apache.hadoop.fs.swift.http.SwiftRestClient.exec(SwiftRestClient.java:1696) at org.apache.hadoop.fs.swift.http.SwiftRestClient.perform(SwiftRestClient.java:1397) at org.apache.hadoop.fs.swift.http.SwiftRestClient.headRequest(SwiftRestClient.java:1026) at org.apache.hadoop.fs.swift.http.SwiftRestClient.createContainer(SwiftRestClient.java:1264) at org.apache.hadoop.fs.swift.http.SwiftRestClient.createDefaultContainer(SwiftRestClient.java:1249) - locked <0x00000007d9580e68> (a org.apache.hadoop.fs.swift.http.SwiftRestClient) at org.apache.hadoop.fs.swift.http.SwiftRestClient.access$1500(SwiftRestClient.java:98) at org.apache.hadoop.fs.swift.http.SwiftRestClient$AuthenticationPost.extractResult(SwiftRestClient.java:1226) at org.apache.hadoop.fs.swift.http.SwiftRestClient$AuthenticationPost.extractResult(SwiftRestClient.java:1095) at org.apache.hadoop.fs.swift.http.SwiftRestClient.perform(SwiftRestClient.java:1418) at org.apache.hadoop.fs.swift.http.SwiftRestClient.authenticate(SwiftRestClient.java:1090) at org.apache.hadoop.fs.swift.http.SwiftRestClient.exec(SwiftRestClient.java:1696) at org.apache.hadoop.fs.swift.http.SwiftRestClient.perform(SwiftRestClient.java:1397) at org.apache.hadoop.fs.swift.http.SwiftRestClient.headRequest(SwiftRestClient.java:1026) at org.apache.hadoop.fs.swift.http.SwiftRestClient.createContainer(SwiftRestClient.java:1264) at org.apache.hadoop.fs.swift.http.SwiftRestClient.createDefaultContainer(SwiftRestClient.java:1249) - locked <0x00000007d9580e68> (a org.apache.hadoop.fs.swift.http.SwiftRestClient) at org.apache.hadoop.fs.swift.http.SwiftRestClient.access$1500(SwiftRestClient.java:98) at org.apache.hadoop.fs.swift.http.SwiftRestClient$AuthenticationPost.extractResult(SwiftRestClient.java:1226) at org.apache.hadoop.fs.swift.http.SwiftRestClient$AuthenticationPost.extractResult(SwiftRestClient.java:1095) at org.apache.hadoop.fs.swift.http.SwiftRestClient.perform(SwiftRestClient.java:1418) at org.apache.hadoop.fs.swift.http.SwiftRestClient.authenticate(SwiftRestClient.java:1090) at org.apache.hadoop.fs.swift.http.TestSwiftRestClient.testAuthenticate(TestSwiftRestClient.java:77) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) "Service Thread " daemon prio=5 tid=0x00007fcf1b813000 nid=0x4d03 runnable [0x0000000000000000]

Hadoop QA added a comment - 19/May/16 16:45 -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 patch 0m 4s HADOOP-11614 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12804938/HADOOP-11614-002.patch JIRA Issue HADOOP-11614 Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9513/console Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.

Wei-Chiu Chuang added a comment - 28/Oct/16 22:09 - edited I want bump the priority of this issue. I think we should really spent sometime to get rid of this ancient third party lib given it's no longer maintained and has security vulnerability. Set target version to 3.0.0-alpha-2 to minimize compatibility issue with Hadoop2.

Akira Ajisaka added a comment - 31/Oct/16 11:01 003 patch: Rebased

Hadoop QA added a comment - 31/Oct/16 11:33 -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 20s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. +1 mvninstall 8m 36s trunk passed +1 compile 0m 15s trunk passed +1 checkstyle 0m 17s trunk passed +1 mvnsite 0m 17s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 0m 22s trunk passed +1 javadoc 0m 12s trunk passed +1 mvninstall 0m 12s the patch passed +1 compile 0m 10s the patch passed -1 javac 0m 10s hadoop-tools_hadoop-openstack generated 7 new + 7 unchanged - 0 fixed = 14 total (was 7) -0 checkstyle 0m 12s hadoop-tools/hadoop-openstack: The patch generated 48 new + 174 unchanged - 85 fixed = 222 total (was 259) +1 mvnsite 0m 14s the patch passed +1 mvneclipse 0m 9s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. -1 findbugs 0m 30s hadoop-tools/hadoop-openstack generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) +1 javadoc 0m 10s the patch passed +1 unit 0m 11s hadoop-openstack in the patch passed. +1 asflicense 0m 15s The patch does not generate ASF License warnings. 13m 54s Reason Tests FindBugs module:hadoop-tools/hadoop-openstack   Possible null pointer dereference of resp in org.apache.hadoop.fs.swift.http.SwiftRestClient.perform(String, URI, SwiftRestClient$HttpRequestProcessor) on exception path Dereferenced at SwiftRestClient.java:resp in org.apache.hadoop.fs.swift.http.SwiftRestClient.perform(String, URI, SwiftRestClient$HttpRequestProcessor) on exception path Dereferenced at SwiftRestClient.java: [line 1423]   Load of known null value in org.apache.hadoop.fs.swift.util.HttpResponseUtils.getResponseBody(HttpResponse) At HttpResponseUtils.java:in org.apache.hadoop.fs.swift.util.HttpResponseUtils.getResponseBody(HttpResponse) At HttpResponseUtils.java: [line 71] Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HADOOP-11614 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12836136/HADOOP-11614-003.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux 681ad83bd1a2 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / f646fe3 Default Java 1.8.0_101 findbugs v3.0.0 javac https://builds.apache.org/job/PreCommit-HADOOP-Build/10930/artifact/patchprocess/diff-compile-javac-hadoop-tools_hadoop-openstack.txt checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/10930/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-openstack.txt findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/10930/artifact/patchprocess/new-findbugs-hadoop-tools_hadoop-openstack.html Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10930/testReport/ modules C: hadoop-tools/hadoop-openstack U: hadoop-tools/hadoop-openstack Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10930/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.

Akira Ajisaka added a comment - 04/Nov/16 10:59 004 patch: Fixed findbugs, javac, and checkstyle warnings.

Hadoop QA added a comment - 04/Nov/16 11:30 -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. +1 mvninstall 7m 35s trunk passed +1 compile 0m 17s trunk passed +1 checkstyle 0m 19s trunk passed +1 mvnsite 0m 18s trunk passed +1 mvneclipse 0m 20s trunk passed +1 findbugs 0m 28s trunk passed +1 javadoc 0m 14s trunk passed +1 mvninstall 0m 12s the patch passed +1 compile 0m 12s the patch passed +1 javac 0m 12s hadoop-tools_hadoop-openstack generated 0 new + 6 unchanged - 1 fixed = 6 total (was 7) -0 checkstyle 0m 11s hadoop-tools/hadoop-openstack: The patch generated 14 new + 132 unchanged - 126 fixed = 146 total (was 258) +1 mvnsite 0m 13s the patch passed +1 mvneclipse 0m 9s the patch passed -1 whitespace 0m 0s The patch has 12 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 0m 29s the patch passed +1 javadoc 0m 11s the patch passed +1 unit 0m 11s hadoop-openstack in the patch passed. +1 asflicense 0m 15s The patch does not generate ASF License warnings. 13m 12s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HADOOP-11614 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12837102/HADOOP-11614-004.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux 25345434c47a 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 19b3779 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/10991/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-openstack.txt whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/10991/artifact/patchprocess/whitespace-eol.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10991/testReport/ modules C: hadoop-tools/hadoop-openstack U: hadoop-tools/hadoop-openstack Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10991/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.

Andrew Wang added a comment - 10/Dec/16 01:20 Ping, what's the plan for this JIRA? It's marked as a blocker, but been sitting idle for a month. If there's no plan to immediately resolve this, I'd like to downgrade the priority. Thanks all.

Akira Ajisaka added a comment - 10/Dec/16 02:02 Thanks Andrew Wang for taking care of this. Updated the target versions.

Akira Ajisaka added a comment - 10/Dec/16 02:06 005 patch: Rebased, and removed trailing whitespaces.

Hadoop QA added a comment - 10/Dec/16 02:37 +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. +1 mvninstall 8m 50s trunk passed +1 compile 0m 14s trunk passed +1 checkstyle 0m 15s trunk passed +1 mvnsite 0m 17s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 0m 28s trunk passed +1 javadoc 0m 13s trunk passed +1 mvninstall 0m 14s the patch passed +1 compile 0m 13s the patch passed +1 javac 0m 13s hadoop-tools_hadoop-openstack generated 0 new + 6 unchanged - 1 fixed = 6 total (was 7) -0 checkstyle 0m 12s hadoop-tools/hadoop-openstack: The patch generated 13 new + 133 unchanged - 126 fixed = 146 total (was 259) +1 mvnsite 0m 16s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. +1 findbugs 0m 35s the patch passed +1 javadoc 0m 11s the patch passed +1 unit 0m 14s hadoop-openstack in the patch passed. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 14m 31s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HADOOP-11614 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12842642/HADOOP-11614-005.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux 83b5649f84cd 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 92a8917 Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/11238/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-openstack.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11238/testReport/ modules C: hadoop-tools/hadoop-openstack U: hadoop-tools/hadoop-openstack Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11238/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.

Akira Ajisaka added a comment - 30/Mar/17 04:17 I cannot work on this issue and fix the test failures because I don't have the environments for testing. Hi Steve Loughran , could you work on this issue? Removing the dependency is marked as blocker for 3.0.0-beta1. If we cannot remove the dependency in hadoop-openstack module, maybe we need to consider dropping the module itself and using the sahara-extra implementation ( https://github.com/openstack/sahara-extra/tree/master/hadoop-swiftfs ) instead.

Hadoop QA added a comment - 30/Mar/17 04:36 +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. +1 mvninstall 12m 44s trunk passed +1 compile 0m 13s trunk passed +1 checkstyle 0m 15s trunk passed +1 mvnsite 0m 15s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 0m 23s trunk passed +1 javadoc 0m 13s trunk passed +1 mvninstall 0m 12s the patch passed +1 compile 0m 11s the patch passed +1 javac 0m 11s hadoop-tools_hadoop-openstack generated 0 new + 6 unchanged - 1 fixed = 6 total (was 7) -0 checkstyle 0m 11s hadoop-tools/hadoop-openstack: The patch generated 14 new + 138 unchanged - 125 fixed = 152 total (was 263) +1 mvnsite 0m 12s the patch passed +1 mvneclipse 0m 9s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. +1 findbugs 0m 28s the patch passed +1 javadoc 0m 9s the patch passed +1 unit 0m 11s hadoop-openstack in the patch passed. +1 asflicense 0m 15s The patch does not generate ASF License warnings. 17m 56s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HADOOP-11614 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12842642/HADOOP-11614-005.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux 077c93fd76e9 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / ceab00a Default Java 1.8.0_121 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/11997/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-openstack.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11997/testReport/ modules C: hadoop-tools/hadoop-openstack U: hadoop-tools/hadoop-openstack Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11997/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.

Steve Loughran added a comment - 30/Mar/17 08:40 I'm not going to get a chance to go near this for 10 days, as I have travel plans. FWIW, the test setup costs ~20c/month, I have accounts with rackspace US and UK. It's not hard to set up. Openstack you can deploy locally; the external ones just give you different problems (auth, throttling,,,)

Akira Ajisaka added a comment - 05/Apr/17 21:09 06 patch Fixed auth failures Some tests fails by SocketException. I'm looking into this. java.net.SocketException: Socket is closed at sun.security.ssl.SSLSocketImpl.checkEOF(SSLSocketImpl.java:1532) at sun.security.ssl.AppInputStream.read(AppInputStream.java:95) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.read(SessionInputBufferImpl.java:208) at org.apache.http.impl.io.ContentLengthInputStream.read(ContentLengthInputStream.java:178) at org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:137) at org.apache.hadoop.fs.swift.http.HttpInputStreamWithRelease.read(HttpInputStreamWithRelease.java:203) at org.apache.hadoop.fs.swift.snative.SwiftNativeInputStream.read(SwiftNativeInputStream.java:177) at org.apache.hadoop.fs.FSInputStream.read(FSInputStream.java:75) at org.apache.hadoop.fs.FSInputStream.readFully(FSInputStream.java:121) at org.apache.hadoop.fs.BufferedFSInputStream.readFully(BufferedFSInputStream.java:113) at org.apache.hadoop.fs.FSDataInputStream.readFully(FSDataInputStream.java:120) at org.apache.hadoop.fs.swift.util.SwiftTestUtils.readBytesToString(SwiftTestUtils.java:425) at org.apache.hadoop.fs.swift.scale.TestWriteManySmallFiles.testScaledWriteThenRead(TestWriteManySmallFiles.java:72)

Hadoop QA added a comment - 05/Apr/17 21:36 +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. +1 mvninstall 12m 54s trunk passed +1 compile 0m 15s trunk passed +1 checkstyle 0m 16s trunk passed +1 mvnsite 0m 17s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 0m 27s trunk passed +1 javadoc 0m 15s trunk passed +1 mvninstall 0m 13s the patch passed +1 compile 0m 12s the patch passed +1 javac 0m 12s hadoop-tools_hadoop-openstack generated 0 new + 5 unchanged - 2 fixed = 5 total (was 7) -0 checkstyle 0m 12s hadoop-tools/hadoop-openstack: The patch generated 13 new + 136 unchanged - 127 fixed = 149 total (was 263) +1 mvnsite 0m 15s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 0m 32s the patch passed +1 javadoc 0m 12s the patch passed +1 unit 0m 13s hadoop-openstack in the patch passed. +1 asflicense 0m 16s The patch does not generate ASF License warnings. 18m 35s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HADOOP-11614 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12862162/HADOOP-11614-006.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux 56bf2b78be7a 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 7d963c4 Default Java 1.8.0_121 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/12034/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-openstack.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12034/testReport/ modules C: hadoop-tools/hadoop-openstack U: hadoop-tools/hadoop-openstack Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12034/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.

Akira Ajisaka added a comment - 14/Apr/17 03:57 Fixed some test failures related to SocketException.

Hadoop QA added a comment - 14/Apr/17 04:17 +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. +1 mvninstall 13m 16s trunk passed +1 compile 0m 15s trunk passed +1 checkstyle 0m 15s trunk passed +1 mvnsite 0m 16s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 0m 24s trunk passed +1 javadoc 0m 14s trunk passed +1 mvninstall 0m 13s the patch passed +1 compile 0m 11s the patch passed +1 javac 0m 11s hadoop-tools_hadoop-openstack generated 0 new + 5 unchanged - 2 fixed = 5 total (was 7) -0 checkstyle 0m 11s hadoop-tools/hadoop-openstack: The patch generated 1 new + 136 unchanged - 127 fixed = 137 total (was 263) +1 mvnsite 0m 14s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 0m 29s the patch passed +1 javadoc 0m 11s the patch passed +1 unit 0m 12s hadoop-openstack in the patch passed. +1 asflicense 0m 16s The patch does not generate ASF License warnings. 18m 39s Subsystem Report/Notes Docker Image:yetus/hadoop:612578f JIRA Issue HADOOP-11614 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12863416/HADOOP-11614-007.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux cf96801c12b3 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 0cab572 Default Java 1.8.0_121 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/12101/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-openstack.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12101/testReport/ modules C: hadoop-tools/hadoop-openstack U: hadoop-tools/hadoop-openstack Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12101/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.

Masatake Iwasaki added a comment - 25/Apr/17 09:39 Thanks for the update, Akira Ajisaka . The patch looks good. Just 1 nit. There is no relevant logger in new httpcore/httpclient. I think we can just remove the line. #crank back on warnings about -1 content length GETs -log4j.logger.org.apache.commons.httpclient.HttpMethodBase=ERROR +log4j.logger.org.apache.http.client.methods.HttpGet=ERROR I ran unit tests including contract tests against Rackspace Cloud Files. Only TestSwiftContractRootDir#testRmNonEmptyRootDirNonRecursive failed but it seems not to be related to the fix since it failed even without the patch. Running org.apache.hadoop.fs.swift.contract.TestSwiftContractRootDir Tests run: 9, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 103.302 sec <<< FAILURE! - in org.apache.hadoop.fs.swift.contract.TestSwiftContractRootDir testRmNonEmptyRootDirNonRecursive(org.apache.hadoop.fs.swift.contract.TestSwiftContractRootDir) Time elapsed: 13.245 sec <<< FAILURE! java.lang.AssertionError: File claims to be a directory /testRmNonEmptyRootDirNonRecursive SwiftFileStatus{ path=swift://testfs.rackspace/testRmNonEmptyRootDirNonRecursive; isDirectory=true; length=0; blocksize=33554432; modification_time=1493112657000} at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.assertTrue(Assert.java:41) at org.junit.Assert.assertFalse(Assert.java:64) at org.apache.hadoop.fs.contract.ContractTestUtils.assertIsFile(ContractTestUtils.java:726) at org.apache.hadoop.fs.contract.ContractTestUtils.assertIsFile(ContractTestUtils.java:715) at org.apache.hadoop.fs.contract.AbstractFSContractTestBase.assertIsFile(AbstractFSContractTestBase.java:316) at org.apache.hadoop.fs.contract.AbstractContractRootDirectoryTest.testRmNonEmptyRootDirNonRecursive(AbstractContractRootDirectoryTest.java:138) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)

Akira Ajisaka added a comment - 26/Apr/17 06:25 Thank Masatake Iwasaki for the review! Updated the patch.

Hadoop QA added a comment - 26/Apr/17 09:57 +1 overall Vote Subsystem Runtime Comment 0 reexec 17m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. +1 mvninstall 13m 40s trunk passed +1 compile 0m 14s trunk passed +1 checkstyle 0m 16s trunk passed +1 mvnsite 0m 16s trunk passed +1 mvneclipse 0m 16s trunk passed +1 findbugs 0m 26s trunk passed +1 javadoc 0m 14s trunk passed +1 mvninstall 0m 12s the patch passed +1 compile 0m 12s the patch passed +1 javac 0m 12s hadoop-tools_hadoop-openstack generated 0 new + 5 unchanged - 2 fixed = 5 total (was 7) -0 checkstyle 0m 11s hadoop-tools/hadoop-openstack: The patch generated 1 new + 136 unchanged - 127 fixed = 137 total (was 263) +1 mvnsite 0m 13s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 0m 28s the patch passed +1 javadoc 0m 10s the patch passed +1 unit 0m 12s hadoop-openstack in the patch passed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 36m 12s Subsystem Report/Notes Docker Image:yetus/hadoop:0ac17dc JIRA Issue HADOOP-11614 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12865076/HADOOP-11614-008.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux a2ffa0b43e1c 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 93fa48f Default Java 1.8.0_131 findbugs v3.1.0-RC1 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/12194/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-openstack.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12194/testReport/ modules C: hadoop-tools/hadoop-openstack U: hadoop-tools/hadoop-openstack Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12194/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.

Masatake Iwasaki added a comment - 27/Apr/17 05:26 +1

Akira Ajisaka added a comment - 27/Apr/17 07:59 Committed this to trunk. Thanks Masatake Iwasaki and Steve Loughran for the review and thanks Brahma Reddy Battula for the contribution.

Brahma Reddy Battula added a comment - 27/Apr/17 09:00 Akira Ajisaka thanks for commit and thanks to others..

Hudson added a comment - 27/Apr/17 12:56 SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11639 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11639/ ) HADOOP-11614 . Remove httpclient dependency from hadoop-openstack. (aajisaka: rev 62579b69a0a294ba1ea14cf76c650b640f89f331) (delete) hadoop-tools/hadoop-openstack/src/main/java/org/apache/hadoop/fs/swift/http/CopyMethod.java (edit) hadoop-tools/hadoop-openstack/src/main/java/org/apache/hadoop/fs/swift/http/SwiftRestClient.java (edit) hadoop-tools/hadoop-openstack/src/test/resources/log4j.properties (edit) hadoop-tools/hadoop-openstack/src/test/java/org/apache/hadoop/fs/swift/http/TestSwiftRestClient.java (add) hadoop-tools/hadoop-openstack/src/main/java/org/apache/hadoop/fs/swift/util/HttpResponseUtils.java (edit) hadoop-tools/hadoop-openstack/src/main/java/org/apache/hadoop/fs/swift/http/HttpInputStreamWithRelease.java (edit) hadoop-tools/hadoop-openstack/src/main/java/org/apache/hadoop/fs/swift/exceptions/SwiftInvalidResponseException.java (edit) hadoop-tools/hadoop-openstack/src/main/java/org/apache/hadoop/fs/swift/exceptions/SwiftThrottledRequestException.java (edit) hadoop-tools/hadoop-openstack/src/main/java/org/apache/hadoop/fs/swift/snative/SwiftNativeFileSystemStore.java (edit) hadoop-tools/hadoop-openstack/src/test/java/org/apache/hadoop/fs/swift/TestSwiftFileSystemPartitionedUploads.java (edit) hadoop-tools/hadoop-openstack/pom.xml (edit) hadoop-tools/hadoop-openstack/src/main/java/org/apache/hadoop/fs/swift/exceptions/SwiftBadRequestException.java (edit) hadoop-tools/hadoop-openstack/src/main/java/org/apache/hadoop/fs/swift/exceptions/SwiftAuthenticationFailedException.java (add) hadoop-tools/hadoop-openstack/src/main/java/org/apache/hadoop/fs/swift/http/CopyRequest.java