Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-2656

Implement a pure c client based on webhdfs

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.0.3-alpha
    • Component/s: webhdfs
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Currently, the implementation of libhdfs is based on JNI. The overhead of JVM seems a little big, and libhdfs can also not be used in the environment without hdfs.
      It seems a good idea to implement a pure c client by wrapping webhdfs. It also can be used to access different version of hdfs.

      1. HDFS-2656.unfinished.patch
        68 kB
        Jing Zhao
      2. HDFS-2656.patch
        188 kB
        Jing Zhao
      3. HDFS-2656.patch
        208 kB
        Jing Zhao
      4. teragen_terasort_teravalidate_performance.png
        39 kB
        Jing Zhao
      5. HDFS-2656.patch
        212 kB
        Jing Zhao

        Issue Links

          Activity

          Hide
          Zhanwei Wang added a comment -

          I am working on a new pure c hdfs client named libchdfs. It has almost the same interface with libhdfs. Any comment are welcomed.

          Show
          Zhanwei Wang added a comment - I am working on a new pure c hdfs client named libchdfs. It has almost the same interface with libhdfs. Any comment are welcomed.
          Hide
          Matteo Bertozzi added a comment -

          nice! have you a repository?
          if you haven't started yet, checkout my draft
          https://github.com/matteobertozzi/webhdfs

          Show
          Matteo Bertozzi added a comment - nice! have you a repository? if you haven't started yet, checkout my draft https://github.com/matteobertozzi/webhdfs
          Hide
          Zhanwei Wang added a comment -

          Great! Matteom. You have done a lot.
          But I intend to implement a pure c lib which has almost the same interface with libhdfs.
          The benefit is
          1) Only few codes of fuse-dfs need to be modified.
          2) Users of libhdfs can easily use new lib instead of libhdfs.

          Show
          Zhanwei Wang added a comment - Great! Matteom. You have done a lot. But I intend to implement a pure c lib which has almost the same interface with libhdfs. The benefit is 1) Only few codes of fuse-dfs need to be modified. 2) Users of libhdfs can easily use new lib instead of libhdfs.
          Hide
          Zhanwei Wang added a comment -

          Hi everyone,
          The code of this jira proposed is almost finished and will available soon. It is more complicated than what I thought and cost me more time. The following is the current status of the code.

          Status update:

          1, finished:
          most functions in hdfs.h of libhdfs are implemented.

          2, on-going:
          function test.
          unit test.
          document

          3, todo:
          kerberos support.
          http proxy support.
          some performance improvement

          Show
          Zhanwei Wang added a comment - Hi everyone, The code of this jira proposed is almost finished and will available soon. It is more complicated than what I thought and cost me more time. The following is the current status of the code. Status update: 1, finished: most functions in hdfs.h of libhdfs are implemented. 2, on-going: function test. unit test. document 3, todo: kerberos support. http proxy support. some performance improvement
          Hide
          donal zang added a comment -

          Hi Zhanwei,
          I'm interested in the c client.
          But one thing I'm worrying about is the performance, since it uses http protocol.
          Have you tested or thought about this?

          Show
          donal zang added a comment - Hi Zhanwei, I'm interested in the c client. But one thing I'm worrying about is the performance, since it uses http protocol. Have you tested or thought about this?
          Hide
          Zhanwei Wang added a comment -

          Hi donal,
          Good question, performance is an important issue and the lib needs to be designed and implemented carefully.

          From lib side, I use libcurl to deal with http protocol and a buffer in the lib to optimize the performance. The same design was also used in our another project and the performance of libcurl is ok.

          For the transmission, http use tcp connection. To read data from server, only the raw data is transfered. To write to server, I use "chunked" transfer encoding, and the overhead is just a small head per chunk.

          For the server side, the performance is depending on the jetty server. In the previous prototype, jetty server or webhdfs had performance problem when I use HTTP1.1 protocol to read data from server, but this problem cannot reproduce when I switch to HTTP1.0 protocol.

          I did simple performance test on the previous prototype, and more performance test work is on the plan.

          Currently, to write to hdfs may still fail under the heavy workload, I am not sure it is a bug of my code or the hdfs, I am working on it (seems not my bug _). The doc is under writing, function test is finished. As soon as I get the permit to open source and finish the doc, you can test yourself. I think it will not take too long.

          Show
          Zhanwei Wang added a comment - Hi donal, Good question, performance is an important issue and the lib needs to be designed and implemented carefully. From lib side, I use libcurl to deal with http protocol and a buffer in the lib to optimize the performance. The same design was also used in our another project and the performance of libcurl is ok. For the transmission, http use tcp connection. To read data from server, only the raw data is transfered. To write to server, I use "chunked" transfer encoding, and the overhead is just a small head per chunk. For the server side, the performance is depending on the jetty server. In the previous prototype, jetty server or webhdfs had performance problem when I use HTTP1.1 protocol to read data from server, but this problem cannot reproduce when I switch to HTTP1.0 protocol. I did simple performance test on the previous prototype, and more performance test work is on the plan. Currently, to write to hdfs may still fail under the heavy workload, I am not sure it is a bug of my code or the hdfs, I am working on it (seems not my bug _ ). The doc is under writing, function test is finished. As soon as I get the permit to open source and finish the doc, you can test yourself. I think it will not take too long.
          Hide
          Suresh Srinivas added a comment -

          The following is the current status of the code

          Can you attach the patch?

          Show
          Suresh Srinivas added a comment - The following is the current status of the code Can you attach the patch?
          Hide
          Zhanwei Wang added a comment -

          hi Suresh Srinivas

          I'm sorry I cannot submit the patch now since I have not got the company's permission to open source this lib.

          I hope I can submit the patch as soon as possible, but I don't know the exact time.

          Show
          Zhanwei Wang added a comment - hi Suresh Srinivas I'm sorry I cannot submit the patch now since I have not got the company's permission to open source this lib. I hope I can submit the patch as soon as possible, but I don't know the exact time.
          Hide
          Suresh Srinivas added a comment -

          The reason I ask is, some of the work done in HDFS-2631 could be used to enable libhdfs2 based on webhdfs. If you do not think you will be able to do this in the near future, I am interested in picking this up.

          Show
          Suresh Srinivas added a comment - The reason I ask is, some of the work done in HDFS-2631 could be used to enable libhdfs2 based on webhdfs. If you do not think you will be able to do this in the near future, I am interested in picking this up.
          Hide
          Kay Yan added a comment -

          I am working on it too.My repo is at https://github.com/yankay/libhadoop.It would be released in a few months.

          Show
          Kay Yan added a comment - I am working on it too.My repo is at https://github.com/yankay/libhadoop.It would be released in a few months.
          Hide
          Kay Yan added a comment -

          libhadoop is not over http like webhdfs, it's over tcp. I would use pure c/c++ to implement the hdfs protocol.
          It may be a little challenge, so before release I would review code and test a lot.

          Show
          Kay Yan added a comment - libhadoop is not over http like webhdfs, it's over tcp. I would use pure c/c++ to implement the hdfs protocol. It may be a little challenge, so before release I would review code and test a lot.
          Hide
          Zhanwei Wang added a comment -

          @yankay

          I have implemented two libs, libhdfs2 is a c lib based on webhdfs and libhdfs3 is a c++ lib based on hdfs protocolbuffer compatible rpc.

          Both libs are used in our project. Since I have not got the company's open source permission, I cannot contribute it to the hadoop.

          The have applied to open source these two libs since February,but until now...

          Show
          Zhanwei Wang added a comment - @yankay I have implemented two libs, libhdfs2 is a c lib based on webhdfs and libhdfs3 is a c++ lib based on hdfs protocolbuffer compatible rpc. Both libs are used in our project. Since I have not got the company's open source permission, I cannot contribute it to the hadoop. The have applied to open source these two libs since February,but until now...
          Hide
          Suresh Srinivas added a comment -

          libhadoop is not over http like webhdfs, it's over tcp. I would use pure c/c++ to implement the hdfs protocol.

          Your comment is then not relevant to this Jira. This Jira specifically talks about WebHDFS based implementation.

          Show
          Suresh Srinivas added a comment - libhadoop is not over http like webhdfs, it's over tcp. I would use pure c/c++ to implement the hdfs protocol. Your comment is then not relevant to this Jira. This Jira specifically talks about WebHDFS based implementation.
          Hide
          Suresh Srinivas added a comment -

          The have applied to open source these two libs since February,but until now...

          Any comment on how soon you can contribute the patch?

          Show
          Suresh Srinivas added a comment - The have applied to open source these two libs since February,but until now... Any comment on how soon you can contribute the patch?
          Hide
          Kay Yan added a comment -

          It would be released in next 2 month.

          Show
          Kay Yan added a comment - It would be released in next 2 month.
          Hide
          Jing Zhao added a comment -

          I'm now working on it based on HDFS-2631.patch by Jaimin. The patch is unfinished and in progress.

          Show
          Jing Zhao added a comment - I'm now working on it based on HDFS-2631 .patch by Jaimin. The patch is unfinished and in progress.
          Hide
          Jing Zhao added a comment -

          Some update on libhdfs2:
          1. Implement most of the functions of original libhdfs based on webhdfs (basic functionalities are based on HDFS-2631.patch by Jaimin);
          2. hdfsCopy, hdfsMove, and hdfsGetDefaultBlockSize are still based on JNI;
          3. hdfsGetHosts, hdfsGetCapacity, and hdfsGetUsed will be implemented in the next step when corresponding webhdfs operations are ready.
          4. Test code is included in the test folder which are derived from the test code in libhdfs.

          Show
          Jing Zhao added a comment - Some update on libhdfs2: 1. Implement most of the functions of original libhdfs based on webhdfs (basic functionalities are based on HDFS-2631 .patch by Jaimin); 2. hdfsCopy, hdfsMove, and hdfsGetDefaultBlockSize are still based on JNI; 3. hdfsGetHosts, hdfsGetCapacity, and hdfsGetUsed will be implemented in the next step when corresponding webhdfs operations are ready. 4. Test code is included in the test folder which are derived from the test code in libhdfs.
          Hide
          Todd Lipcon added a comment -

          Can you please comment on performance of this implementation vs the libhdfs in trunk?

          I am uncomfortable calling this "libhdfs2" if the performance is not up to par with the current "libhdfs1". One of the major reasons people use libhdfs is that they're writing a high-performance app, and I would be really surprised if this HTTP-based client can match the performance of the JNI-based one, especially when short-circuit read is enabled.

          An additional concern is that webhdfs doesn't seem to return any checksum information with its transfers. So, a client using this C library no longer has the benefits of checksumming that HDFS provides.

          So while I see there are some uses for this, I don't think it's viable as a replacement. Perhaps better to describe it as "libwebhdfs" or something?

          The "libhdfs3" work on the other hand sounds useful for a variety of applications, and should have better performance and memory footprint while not giving up features like checksumming, etc. Any word on when that would be available?

          Show
          Todd Lipcon added a comment - Can you please comment on performance of this implementation vs the libhdfs in trunk? I am uncomfortable calling this "libhdfs2" if the performance is not up to par with the current "libhdfs1". One of the major reasons people use libhdfs is that they're writing a high-performance app, and I would be really surprised if this HTTP-based client can match the performance of the JNI-based one, especially when short-circuit read is enabled. An additional concern is that webhdfs doesn't seem to return any checksum information with its transfers. So, a client using this C library no longer has the benefits of checksumming that HDFS provides. So while I see there are some uses for this, I don't think it's viable as a replacement . Perhaps better to describe it as "libwebhdfs" or something? The "libhdfs3" work on the other hand sounds useful for a variety of applications, and should have better performance and memory footprint while not giving up features like checksumming, etc. Any word on when that would be available?
          Hide
          Zhanwei Wang added a comment -

          @Jing Zhao

          Hi, I just had a quick look at your patch, and I had something worry about:

          1) In hdfsOpenFile, it seems not really open on a hdfs file, just create a handle in libhdfs2. If open for read, it cannot check the error such as file not existing. If open for write, the libhdfs2 did not hold the lease, that means other client such java client and libhdfs client still alos can open file for write. It is big semantic difference with libhdfs.

          2) For each read/write, create new connection to namenode and datanode and close the connection after the operation, It seems a performance issue.

          B. Todd Burruss
          I do not thing libhdfs2 is a replacement of libhdfs since:
          1) The performance of this patch seems a issue. In my implementation, I setup only one http connection for a opened file and keep the connection until file closed. The performance is about 5%~10% slower than java client(without short-circuit read), I did not compare it with the libhdfs. I think the overhead is on http server and data locality.
          2) In my implementation, hdfsFlush is implemented as closing and reopening the file, has different semantic with libhdfs. and in this patch, the semantic difference of oppen is bigger.

          I use libhdfs2 since jvm use too many memory and I also want better performance. Currently I have moved to libhdfs3, that is a really replacement of libhdfs

          About the checksum of libhdfs2, I don't think that is a problem, since http server use java client to read data and already verify the checksum, and http connection is over tcp, the possibility of read unexpected data is very small.

          Show
          Zhanwei Wang added a comment - @Jing Zhao Hi, I just had a quick look at your patch, and I had something worry about: 1) In hdfsOpenFile, it seems not really open on a hdfs file, just create a handle in libhdfs2. If open for read, it cannot check the error such as file not existing. If open for write, the libhdfs2 did not hold the lease, that means other client such java client and libhdfs client still alos can open file for write. It is big semantic difference with libhdfs. 2) For each read/write, create new connection to namenode and datanode and close the connection after the operation, It seems a performance issue. B. Todd Burruss I do not thing libhdfs2 is a replacement of libhdfs since: 1) The performance of this patch seems a issue. In my implementation, I setup only one http connection for a opened file and keep the connection until file closed. The performance is about 5%~10% slower than java client(without short-circuit read), I did not compare it with the libhdfs. I think the overhead is on http server and data locality. 2) In my implementation, hdfsFlush is implemented as closing and reopening the file, has different semantic with libhdfs. and in this patch, the semantic difference of oppen is bigger. I use libhdfs2 since jvm use too many memory and I also want better performance. Currently I have moved to libhdfs3, that is a really replacement of libhdfs About the checksum of libhdfs2, I don't think that is a problem, since http server use java client to read data and already verify the checksum, and http connection is over tcp, the possibility of read unexpected data is very small.
          Hide
          Jing Zhao added a comment -

          Todd and Zhanwei, thanks very much for the feedback! So for the checksum, since the webhdfs provides the GETFILECHECKSUM operation, I guess we can at least use it for checksumming after finishing reading the whole file. As you guys pointed out, the current patch still needs a lot of improvement and measurement in performance. Also, I agree that the openfile and further corresponding read/write can have a better implementation to keep consistent with libhdfs. I'm now working on it and will upload patch for that.

          Show
          Jing Zhao added a comment - Todd and Zhanwei, thanks very much for the feedback! So for the checksum, since the webhdfs provides the GETFILECHECKSUM operation, I guess we can at least use it for checksumming after finishing reading the whole file. As you guys pointed out, the current patch still needs a lot of improvement and measurement in performance. Also, I agree that the openfile and further corresponding read/write can have a better implementation to keep consistent with libhdfs. I'm now working on it and will upload patch for that.
          Hide
          Jing Zhao added a comment -

          And for the name, because the patch is still at an early stage, we currently call it "libhdfs2" to distinguish with libhdfs. We may decide its name later.

          Show
          Jing Zhao added a comment - And for the name, because the patch is still at an early stage, we currently call it "libhdfs2" to distinguish with libhdfs. We may decide its name later.
          Hide
          Eli Collins added a comment -

          Hi Jing Zhao,

          What's your use case? Do you have applications that need this?
          How about "libwebhdfs" per Todd's suggestion?

          Thanks,
          Eli

          Show
          Eli Collins added a comment - Hi Jing Zhao, What's your use case? Do you have applications that need this? How about "libwebhdfs" per Todd's suggestion? Thanks, Eli
          Hide
          Suresh Srinivas added a comment -

          Some comments:

          I am uncomfortable calling this "libhdfs2" if the performance is not up to par with the current "libhdfs1".

          I understand calling it libwebhdfs for better clarity. But the above reason, based on performance does not make much sense to me.

          @Eli What's your use case?

          I am very surprised by this. You made a case of this in the related jira HDFS-2631, here and here. I know of several people building there own libraries for webhdfs interfaces. This would provide one in Hadoop that they can use. This will also help us understand the performance differences between JNI based implementation vs webhdfs.

          I agree libhdfs3 that directly uses protobuf interfaces is the most useful. But DFSClient today is a very think client with all kinds of sophisticated logic put into it, that uses both RPC and DataTransferProtocol. As we speak it keeps becoming more sophisticated (see HDFS-3672). I am not sure of seeing a complete, robust implementation in C in the near term. I also do not think c implementation can keep up with the changes we keep making in DFSClient.

          Show
          Suresh Srinivas added a comment - Some comments: I am uncomfortable calling this "libhdfs2" if the performance is not up to par with the current "libhdfs1". I understand calling it libwebhdfs for better clarity. But the above reason, based on performance does not make much sense to me. @Eli What's your use case? I am very surprised by this. You made a case of this in the related jira HDFS-2631 , here and here . I know of several people building there own libraries for webhdfs interfaces. This would provide one in Hadoop that they can use. This will also help us understand the performance differences between JNI based implementation vs webhdfs. I agree libhdfs3 that directly uses protobuf interfaces is the most useful. But DFSClient today is a very think client with all kinds of sophisticated logic put into it, that uses both RPC and DataTransferProtocol. As we speak it keeps becoming more sophisticated (see HDFS-3672 ). I am not sure of seeing a complete, robust implementation in C in the near term. I also do not think c implementation can keep up with the changes we keep making in DFSClient.
          Hide
          Eli Collins added a comment -

          Suresh, I just asked him what his use case and applications were because I'm curious =) I'm not opposed to this change, per all my discussion you pointed out I think a C-binding for the WEBHDFS API is useful, in fact I think doing this in libhdfs makes more sense than fuse-dfs.

          Show
          Eli Collins added a comment - Suresh, I just asked him what his use case and applications were because I'm curious =) I'm not opposed to this change, per all my discussion you pointed out I think a C-binding for the WEBHDFS API is useful, in fact I think doing this in libhdfs makes more sense than fuse-dfs.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Hi Jing,

          Thanks for picking this up. I just have checked the read code.

          Currently, hdfsOpenFile initializes some data structure, each hdfsRead creates an individual URL connection (i.e. read n times have n URL connections) and then hdfsCloseFile free the memory. hdfsOpenFile, hdfsRead and hdfsCloseFile should be implemented with a single URL connection. I am not familiar with libcurl, it seems that it could be done by curl_easy_pause:

          • hdfsOpenFile connects to the web server and then pause;
          • hdfsRead let curl fill up the buffer and then pause;
          • hdfsCloseFile closes the connection and free the memory.

          Also, the are unnecessary memcpy in the current implementation. It creates Response in launchCMD(..) and allocates memory in writefunc but not using the buffer provided by the user. In hdfsRead, it copies the data from Response to user buffer. I think we may be able to avoid the copying.

          Show
          Tsz Wo Nicholas Sze added a comment - Hi Jing, Thanks for picking this up. I just have checked the read code. Currently, hdfsOpenFile initializes some data structure, each hdfsRead creates an individual URL connection (i.e. read n times have n URL connections) and then hdfsCloseFile free the memory. hdfsOpenFile, hdfsRead and hdfsCloseFile should be implemented with a single URL connection. I am not familiar with libcurl, it seems that it could be done by curl_easy_pause : hdfsOpenFile connects to the web server and then pause; hdfsRead let curl fill up the buffer and then pause; hdfsCloseFile closes the connection and free the memory. Also, the are unnecessary memcpy in the current implementation. It creates Response in launchCMD(..) and allocates memory in writefunc but not using the buffer provided by the user. In hdfsRead, it copies the data from Response to user buffer. I think we may be able to avoid the copying.
          Hide
          Jing Zhao added a comment -

          Some update on the libwebhdfs. The main change is trying to keep the same writing semantic with libhdfs (thanks to Zhanwei for pointing that out), i.e., when a client opens a file for writing/appending, before the client closes the file, other clients should not be able to open the same file for writing. This is achieved by maintain a single http connection with the corresponding datanode between the open and close operation. Also addressed part of Nicholas's comments – try to get rid of some of unnecessary memory copying.

          For the performance measurement, before we directly compare the performance between libhdfs and libwebhdfs, we first run teragen, terasort, teravalidate in a 3-node mini cluster (data size 100,000,000), and compare the performance between using DFSClient and WebHdfs. The measurement result is also attached. It seems like the main performance bottleneck for webhdfs is in reading (teravalidate), which is >3 times more than DFSClient. This is maybe because currently webhdfs uses a datanode as a proxy node for reading data even if it is across block boundaries.

          So in the next step, my work will focus on 1) test/fix/improve current code, and 2) to develop a smarter reading mechanism in the client side (i.e., to identify the block locations for a large file in the client side), and 3) to improve client reading performance by decreasing the number of times of http connection creation.

          Waiting for your guys' comments!

          Show
          Jing Zhao added a comment - Some update on the libwebhdfs. The main change is trying to keep the same writing semantic with libhdfs (thanks to Zhanwei for pointing that out), i.e., when a client opens a file for writing/appending, before the client closes the file, other clients should not be able to open the same file for writing. This is achieved by maintain a single http connection with the corresponding datanode between the open and close operation. Also addressed part of Nicholas's comments – try to get rid of some of unnecessary memory copying. For the performance measurement, before we directly compare the performance between libhdfs and libwebhdfs, we first run teragen, terasort, teravalidate in a 3-node mini cluster (data size 100,000,000), and compare the performance between using DFSClient and WebHdfs. The measurement result is also attached. It seems like the main performance bottleneck for webhdfs is in reading (teravalidate), which is >3 times more than DFSClient. This is maybe because currently webhdfs uses a datanode as a proxy node for reading data even if it is across block boundaries. So in the next step, my work will focus on 1) test/fix/improve current code, and 2) to develop a smarter reading mechanism in the client side (i.e., to identify the block locations for a large file in the client side), and 3) to improve client reading performance by decreasing the number of times of http connection creation. Waiting for your guys' comments!
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Hi Jing, if the patch is ready, please try submitting it to see if Jenkins is able to work with it. Thanks.

          Show
          Tsz Wo Nicholas Sze added a comment - Hi Jing, if the patch is ready, please try submitting it to see if Jenkins is able to work with it. Thanks.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12542013/teragen_terasort_teravalidate_performance.png
          against trunk revision .

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3152//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12542013/teragen_terasort_teravalidate_performance.png against trunk revision . -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3152//console This message is automatically generated.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Hi Jing, Jenkins always picks up the latest file. So it tried to download teragen_terasort_teravalidate_performance.png last time. Please simply re-post your patch.

          Show
          Tsz Wo Nicholas Sze added a comment - Hi Jing, Jenkins always picks up the latest file. So it tried to download teragen_terasort_teravalidate_performance.png last time. Please simply re-post your patch.
          Hide
          Jing Zhao added a comment -

          Include the FindJansson.cmake in the patch, and also add a require.libwebhdfs flag to indicate whether or not to compile libwebhdfs in the mvn file.

          Show
          Jing Zhao added a comment - Include the FindJansson.cmake in the patch, and also add a require.libwebhdfs flag to indicate whether or not to compile libwebhdfs in the mvn file.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12544145/HDFS-2656.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 6 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3158//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3158//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12544145/HDFS-2656.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3158//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3158//console This message is automatically generated.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Although Jenkins +1'ed in the previous message but it actually was not yet able to build libwebhdfs. I filed HDFS-3909 for it.

          Show
          Tsz Wo Nicholas Sze added a comment - Although Jenkins +1'ed in the previous message but it actually was not yet able to build libwebhdfs. I filed HDFS-3909 for it.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          +1 patch looks good.

          Show
          Tsz Wo Nicholas Sze added a comment - +1 patch looks good.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          I have committed this. Thanks, Jaimin and Jing!

          Show
          Tsz Wo Nicholas Sze added a comment - I have committed this. Thanks, Jaimin and Jing!
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #2712 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2712/)
          HDFS-2656. Add libwebhdfs, a pure C client based on WebHDFS. Contributed by Jaimin D Jetly and Jing Zhao (Revision 1382836)

          Result = SUCCESS
          szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1382836
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/pom.xml
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/CMakeLists.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/CMakeLists.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/resources
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/resources/FindJansson.cmake
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/exception.c
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/exception.h
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/expect.h
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_http_client.c
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_http_client.h
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_http_query.c
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_http_query.h
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_jni.c
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_json_parser.c
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_json_parser.h
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_web.c
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/jni_helper.c
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/jni_helper.h
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/test_libwebhdfs_multi_write.c
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/test_libwebhdfs_ops.c
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/test_libwebhdfs_read.c
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/test_libwebhdfs_threaded.c
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/test_libwebhdfs_write.c
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/test_read_bm.c
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/webhdfs.h
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2712 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2712/ ) HDFS-2656 . Add libwebhdfs, a pure C client based on WebHDFS. Contributed by Jaimin D Jetly and Jing Zhao (Revision 1382836) Result = SUCCESS szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1382836 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/pom.xml /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/CMakeLists.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/CMakeLists.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/resources /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/resources/FindJansson.cmake /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/exception.c /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/exception.h /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/expect.h /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_http_client.c /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_http_client.h /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_http_query.c /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_http_query.h /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_jni.c /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_json_parser.c /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_json_parser.h /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_web.c /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/jni_helper.c /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/jni_helper.h /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/test_libwebhdfs_multi_write.c /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/test_libwebhdfs_ops.c /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/test_libwebhdfs_read.c /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/test_libwebhdfs_threaded.c /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/test_libwebhdfs_write.c /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/test_read_bm.c /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/webhdfs.h
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #2775 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2775/)
          HDFS-2656. Add libwebhdfs, a pure C client based on WebHDFS. Contributed by Jaimin D Jetly and Jing Zhao (Revision 1382836)

          Result = SUCCESS
          szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1382836
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/pom.xml
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/CMakeLists.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/CMakeLists.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/resources
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/resources/FindJansson.cmake
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/exception.c
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/exception.h
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/expect.h
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_http_client.c
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_http_client.h
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_http_query.c
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_http_query.h
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_jni.c
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_json_parser.c
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_json_parser.h
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_web.c
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/jni_helper.c
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/jni_helper.h
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/test_libwebhdfs_multi_write.c
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/test_libwebhdfs_ops.c
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/test_libwebhdfs_read.c
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/test_libwebhdfs_threaded.c
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/test_libwebhdfs_write.c
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/test_read_bm.c
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/webhdfs.h
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2775 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2775/ ) HDFS-2656 . Add libwebhdfs, a pure C client based on WebHDFS. Contributed by Jaimin D Jetly and Jing Zhao (Revision 1382836) Result = SUCCESS szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1382836 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/pom.xml /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/CMakeLists.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/CMakeLists.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/resources /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/resources/FindJansson.cmake /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/exception.c /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/exception.h /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/expect.h /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_http_client.c /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_http_client.h /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_http_query.c /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_http_query.h /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_jni.c /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_json_parser.c /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_json_parser.h /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_web.c /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/jni_helper.c /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/jni_helper.h /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/test_libwebhdfs_multi_write.c /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/test_libwebhdfs_ops.c /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/test_libwebhdfs_read.c /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/test_libwebhdfs_threaded.c /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/test_libwebhdfs_write.c /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/test_read_bm.c /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/webhdfs.h
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #2736 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2736/)
          HDFS-2656. Add libwebhdfs, a pure C client based on WebHDFS. Contributed by Jaimin D Jetly and Jing Zhao (Revision 1382836)

          Result = FAILURE
          szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1382836
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/pom.xml
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/CMakeLists.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/CMakeLists.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/resources
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/resources/FindJansson.cmake
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/exception.c
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/exception.h
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/expect.h
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_http_client.c
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_http_client.h
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_http_query.c
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_http_query.h
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_jni.c
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_json_parser.c
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_json_parser.h
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_web.c
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/jni_helper.c
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/jni_helper.h
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/test_libwebhdfs_multi_write.c
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/test_libwebhdfs_ops.c
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/test_libwebhdfs_read.c
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/test_libwebhdfs_threaded.c
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/test_libwebhdfs_write.c
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/test_read_bm.c
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/webhdfs.h
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2736 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2736/ ) HDFS-2656 . Add libwebhdfs, a pure C client based on WebHDFS. Contributed by Jaimin D Jetly and Jing Zhao (Revision 1382836) Result = FAILURE szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1382836 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/pom.xml /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/CMakeLists.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/CMakeLists.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/resources /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/resources/FindJansson.cmake /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/exception.c /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/exception.h /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/expect.h /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_http_client.c /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_http_client.h /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_http_query.c /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_http_query.h /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_jni.c /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_json_parser.c /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_json_parser.h /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_web.c /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/jni_helper.c /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/jni_helper.h /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/test_libwebhdfs_multi_write.c /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/test_libwebhdfs_ops.c /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/test_libwebhdfs_read.c /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/test_libwebhdfs_threaded.c /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/test_libwebhdfs_write.c /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/test_read_bm.c /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/webhdfs.h
          Hide
          Todd Lipcon added a comment -

          Hi,

          One question about this patch: one of the stated benefits of libwebhdfs is that it wouldn't require a JVM to be loaded, and thus the clients would use less memory. But I see lots of JNI code in the patch. So was this goal not attained? If so, what's the benefit of libwebhdfs over just using the existing libhdfs with a webhdfs:// URI?

          Show
          Todd Lipcon added a comment - Hi, One question about this patch: one of the stated benefits of libwebhdfs is that it wouldn't require a JVM to be loaded, and thus the clients would use less memory. But I see lots of JNI code in the patch. So was this goal not attained? If so, what's the benefit of libwebhdfs over just using the existing libhdfs with a webhdfs:// URI?
          Hide
          Jing Zhao added a comment -

          Thanks for the comment Todd! So right now we keep JNI temporarily for the following issues:
          1. Load configuration settings for Namenode hostname;
          2. hdfsCopy.
          3. hfdfMove.

          The current implementation is to load JVM in a lazy way. We're trying to avoid using JNI code in our next step. However, hdfsCopy/hdfsMove is not supported by webhdfs now. It seems not simple to re-implement them in the C library. Any suggestion about it?

          Show
          Jing Zhao added a comment - Thanks for the comment Todd! So right now we keep JNI temporarily for the following issues: 1. Load configuration settings for Namenode hostname; 2. hdfsCopy. 3. hfdfMove. The current implementation is to load JVM in a lazy way. We're trying to avoid using JNI code in our next step. However, hdfsCopy/hdfsMove is not supported by webhdfs now. It seems not simple to re-implement them in the C library. Any suggestion about it?
          Hide
          Colin Patrick McCabe added a comment -

          I'm glad we managed to get this merged. Good job, Jing!

          I think there's some refactoring we can do to improve it further. Please take a look at HDFS-3916 when you get a chance.

          Also re: JNI-- I think we might be better off without it. One of the big benefits of webhdfs is that you can update the server components without changing the software on the clients. Because it's a stable REST API, everything will "just work"-- no need to roll out new JAR files to every client. That benefit is lost if we start using JNI in libwebhdfs.

          We might be able to read the XML preference files with an XML parsing library, and determine the default NameNode and port this way. After all, Hadoop preference files are just XML files.

          Show
          Colin Patrick McCabe added a comment - I'm glad we managed to get this merged. Good job, Jing! I think there's some refactoring we can do to improve it further. Please take a look at HDFS-3916 when you get a chance. Also re: JNI-- I think we might be better off without it. One of the big benefits of webhdfs is that you can update the server components without changing the software on the clients. Because it's a stable REST API, everything will "just work"-- no need to roll out new JAR files to every client. That benefit is lost if we start using JNI in libwebhdfs. We might be able to read the XML preference files with an XML parsing library, and determine the default NameNode and port this way. After all, Hadoop preference files are just XML files.
          Hide
          Jing Zhao added a comment -

          Colin, thanks very much for the comment and the cleanup!

          I agree we'd better get rid of the JNI code for libwebhdfs. We also think about the possible solution in which libwebdhfs uses its own preference files to determine the default NamaNode hostname and port. I will include that in my pipeline.

          For hdfsCopy/hdfsMove, any comments about their implementation without JNI?

          Show
          Jing Zhao added a comment - Colin, thanks very much for the comment and the cleanup! I agree we'd better get rid of the JNI code for libwebhdfs. We also think about the possible solution in which libwebdhfs uses its own preference files to determine the default NamaNode hostname and port. I will include that in my pipeline. For hdfsCopy/hdfsMove, any comments about their implementation without JNI?
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1162 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1162/)
          HDFS-2656. Add libwebhdfs, a pure C client based on WebHDFS. Contributed by Jaimin D Jetly and Jing Zhao (Revision 1382836)

          Result = FAILURE
          szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1382836
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/pom.xml
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/CMakeLists.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/CMakeLists.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/resources
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/resources/FindJansson.cmake
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/exception.c
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/exception.h
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/expect.h
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_http_client.c
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_http_client.h
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_http_query.c
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_http_query.h
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_jni.c
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_json_parser.c
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_json_parser.h
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_web.c
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/jni_helper.c
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/jni_helper.h
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/test_libwebhdfs_multi_write.c
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/test_libwebhdfs_ops.c
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/test_libwebhdfs_read.c
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/test_libwebhdfs_threaded.c
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/test_libwebhdfs_write.c
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/test_read_bm.c
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/webhdfs.h
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1162 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1162/ ) HDFS-2656 . Add libwebhdfs, a pure C client based on WebHDFS. Contributed by Jaimin D Jetly and Jing Zhao (Revision 1382836) Result = FAILURE szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1382836 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/pom.xml /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/CMakeLists.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/CMakeLists.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/resources /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/resources/FindJansson.cmake /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/exception.c /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/exception.h /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/expect.h /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_http_client.c /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_http_client.h /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_http_query.c /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_http_query.h /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_jni.c /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_json_parser.c /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_json_parser.h /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_web.c /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/jni_helper.c /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/jni_helper.h /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/test_libwebhdfs_multi_write.c /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/test_libwebhdfs_ops.c /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/test_libwebhdfs_read.c /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/test_libwebhdfs_threaded.c /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/test_libwebhdfs_write.c /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/test_read_bm.c /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/webhdfs.h
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1193 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1193/)
          HDFS-2656. Add libwebhdfs, a pure C client based on WebHDFS. Contributed by Jaimin D Jetly and Jing Zhao (Revision 1382836)

          Result = SUCCESS
          szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1382836
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/pom.xml
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/CMakeLists.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/CMakeLists.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/resources
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/resources/FindJansson.cmake
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/exception.c
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/exception.h
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/expect.h
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_http_client.c
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_http_client.h
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_http_query.c
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_http_query.h
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_jni.c
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_json_parser.c
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_json_parser.h
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_web.c
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/jni_helper.c
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/jni_helper.h
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/test_libwebhdfs_multi_write.c
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/test_libwebhdfs_ops.c
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/test_libwebhdfs_read.c
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/test_libwebhdfs_threaded.c
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/test_libwebhdfs_write.c
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/test_read_bm.c
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/webhdfs.h
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1193 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1193/ ) HDFS-2656 . Add libwebhdfs, a pure C client based on WebHDFS. Contributed by Jaimin D Jetly and Jing Zhao (Revision 1382836) Result = SUCCESS szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1382836 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/pom.xml /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/CMakeLists.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/CMakeLists.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/resources /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/resources/FindJansson.cmake /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/exception.c /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/exception.h /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/expect.h /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_http_client.c /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_http_client.h /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_http_query.c /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_http_query.h /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_jni.c /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_json_parser.c /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_json_parser.h /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_web.c /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/jni_helper.c /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/jni_helper.h /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/test_libwebhdfs_multi_write.c /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/test_libwebhdfs_ops.c /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/test_libwebhdfs_read.c /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/test_libwebhdfs_threaded.c /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/test_libwebhdfs_write.c /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/test_read_bm.c /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/webhdfs.h
          Hide
          Colin Patrick McCabe added a comment -

          For hdfsCopy/hdfsMove, any comments about their implementation without JNI?

          In the JNI-based libhdfs, hdfsCopy calls FileUtil#copy. This method basically just does what you would expect-- it opens an input stream for the input file(s), and copies the bytes to the output file(s).

          Probably the most complicated part of FileUtil#copy is that it works recursively when used on directories. However, there's no special atomicity semantics when FileUtil#copy copies a directory-- it just iterates through the files in a directory and copies what it can. If the directory is updated while it's iterating, that may or may not be reflected in the copy. hdfsMove basically just invokes FileUtil#copy with deleteSource = true.

          So I think we should implement these methods in libwebhdfs using the methods which are already present.

          Show
          Colin Patrick McCabe added a comment - For hdfsCopy/hdfsMove, any comments about their implementation without JNI? In the JNI-based libhdfs, hdfsCopy calls FileUtil#copy . This method basically just does what you would expect-- it opens an input stream for the input file(s), and copies the bytes to the output file(s). Probably the most complicated part of FileUtil#copy is that it works recursively when used on directories. However, there's no special atomicity semantics when FileUtil#copy copies a directory-- it just iterates through the files in a directory and copies what it can. If the directory is updated while it's iterating, that may or may not be reflected in the copy. hdfsMove basically just invokes FileUtil#copy with deleteSource = true. So I think we should implement these methods in libwebhdfs using the methods which are already present.
          Hide
          Suresh Srinivas added a comment -

          We might be able to read the XML preference files with an XML parsing library, and determine the default NameNode and port this way. After all, Hadoop preference files are just XML files.

          The issue here is not XML parsing. The issue is finding the XML files that are loaded from classpath and jar files.

          BTW instead of HDFS-3916, providing some of the feedback that motivated that jira as comments on this jira would have been better. That way a person working on a jira understands what is required of the work they are doing. Also a work gets in with some level of completeness.

          Show
          Suresh Srinivas added a comment - We might be able to read the XML preference files with an XML parsing library, and determine the default NameNode and port this way. After all, Hadoop preference files are just XML files. The issue here is not XML parsing. The issue is finding the XML files that are loaded from classpath and jar files. BTW instead of HDFS-3916 , providing some of the feedback that motivated that jira as comments on this jira would have been better. That way a person working on a jira understands what is required of the work they are doing. Also a work gets in with some level of completeness.
          Hide
          Todd Lipcon added a comment -

          Hey Suresh. Not speaking for Colin, but regarding my comment above: I thought this JIRA was committed rather abruptly. Since the patch was quite big (6kloc), and there had only been one or two small review comments by Nicholas above (in which he said "I just have checked the read code."), I thought the patch was still preliminary and there would be more time to review.

          Since it's in contrib, I don't think we need to revert it for more review, but I do think it makes sense to continue to improve and address people's comments in a follow-up JIRA. Do you think it would be better to revert the patch and address the commentary?

          Maybe we should have some kind of policy that, for large patches like this, committers should wait a day or two between giving a +1 and committing, in case others want to review? I don't think it's really necessary, but maybe it's better than post-commit review in a followup, or asking for a revert in the same JIRA.

          Show
          Todd Lipcon added a comment - Hey Suresh. Not speaking for Colin, but regarding my comment above: I thought this JIRA was committed rather abruptly. Since the patch was quite big (6kloc), and there had only been one or two small review comments by Nicholas above (in which he said "I just have checked the read code."), I thought the patch was still preliminary and there would be more time to review. Since it's in contrib, I don't think we need to revert it for more review, but I do think it makes sense to continue to improve and address people's comments in a follow-up JIRA. Do you think it would be better to revert the patch and address the commentary? Maybe we should have some kind of policy that, for large patches like this, committers should wait a day or two between giving a +1 and committing, in case others want to review? I don't think it's really necessary, but maybe it's better than post-commit review in a followup, or asking for a revert in the same JIRA.
          Hide
          Colin Patrick McCabe added a comment -

          The issue here is not XML parsing. The issue is finding the XML files that are loaded from classpath and jar files.

          I think the CLASSPATH environment variable should have a list of directories we should search for XML files, right? Anyway, Jing has filed HDFS-3917 to discuss this issue, perhaps we should continue our conversation there.

          I also agree that it might have been nice to do some of the refactoring before committing this. However, since this is new code (nobody is using it yet), and not compiled by default, we do have some ability to fix things before it becomes production code.

          Show
          Colin Patrick McCabe added a comment - The issue here is not XML parsing. The issue is finding the XML files that are loaded from classpath and jar files. I think the CLASSPATH environment variable should have a list of directories we should search for XML files, right? Anyway, Jing has filed HDFS-3917 to discuss this issue, perhaps we should continue our conversation there. I also agree that it might have been nice to do some of the refactoring before committing this. However, since this is new code (nobody is using it yet), and not compiled by default, we do have some ability to fix things before it becomes production code.
          Hide
          Suresh Srinivas added a comment -

          Hey Suresh. Not speaking for Colin, but regarding my comment above: I thought this JIRA was committed rather abruptly.

          Todd, the patch in its various forms has been making progress for a very long time. I agree that this is a big patch. Anybody who wants to review could just post a comment saying can please hold this for review.

          But agreed, perhaps, before committing we could have posted a comment saying does anyone have any comments.

          Also one could come back and comment on this jira with further comments. We could then create further jiras. Given Jing has spent quite a lot of time on it and his expertise is more around java than c, addressing comments was a nice way for him to understand this part of the code better. But that said, he could look at cleanups to understand what is expected in these modules.

          Maybe we should have some kind of policy that, for large patches like this, committers should wait a day or two between giving a +1 and committing, in case others want to review? I don't think it's really necessary, but maybe it's better than post-commit review in a followup, or asking for a revert in the same JIRA.

          +1 for the idea. We do not need a rule for this. Committer could just follow this in practice. Also interested folks could just post a comment to hold the commit until review (in a reasonable timeframe).

          Show
          Suresh Srinivas added a comment - Hey Suresh. Not speaking for Colin, but regarding my comment above: I thought this JIRA was committed rather abruptly. Todd, the patch in its various forms has been making progress for a very long time. I agree that this is a big patch. Anybody who wants to review could just post a comment saying can please hold this for review. But agreed, perhaps, before committing we could have posted a comment saying does anyone have any comments. Also one could come back and comment on this jira with further comments. We could then create further jiras. Given Jing has spent quite a lot of time on it and his expertise is more around java than c, addressing comments was a nice way for him to understand this part of the code better. But that said, he could look at cleanups to understand what is expected in these modules. Maybe we should have some kind of policy that, for large patches like this, committers should wait a day or two between giving a +1 and committing, in case others want to review? I don't think it's really necessary, but maybe it's better than post-commit review in a followup, or asking for a revert in the same JIRA. +1 for the idea. We do not need a rule for this. Committer could just follow this in practice. Also interested folks could just post a comment to hold the commit until review (in a reasonable timeframe).
          Hide
          Eli Collins added a comment -

          I'm also a little surprised this was checked in w/o it actually building (HDFS-3909).

          Show
          Eli Collins added a comment - I'm also a little surprised this was checked in w/o it actually building ( HDFS-3909 ).
          Hide
          Suresh Srinivas added a comment -

          I'm also a little surprised this was checked in w/o it actually building (HDFS-3909).

          It is not built by Jenkins because of a need for a library. It has been manually tested as you see in various benchmarks etc. posted.

          Show
          Suresh Srinivas added a comment - I'm also a little surprised this was checked in w/o it actually building ( HDFS-3909 ). It is not built by Jenkins because of a need for a library. It has been manually tested as you see in various benchmarks etc. posted.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          ... I thought this JIRA was committed rather abruptly. Since the patch was quite big (6kloc), and there had only been one or two small review comments by Nicholas above (in which he said "I just have checked the read code."), I thought the patch was still preliminary and there would be more time to review.

          Hi Todd, my "I just have checked the read code." was posted on Aug 20. After that Jing uploaded a new patch addressed my comment and other problems on Aug 22. Jing uploaded another patch with minor change on Sep 6. I spent a few days reviewing and testing the patch and then I +1'ed and committed it on Sep 10. Quite a few people including you provided comments on the earlier patches but no additional comments were posted. I believed there was enough time for people to review it.

          Show
          Tsz Wo Nicholas Sze added a comment - ... I thought this JIRA was committed rather abruptly. Since the patch was quite big (6kloc), and there had only been one or two small review comments by Nicholas above (in which he said "I just have checked the read code."), I thought the patch was still preliminary and there would be more time to review. Hi Todd, my "I just have checked the read code." was posted on Aug 20. After that Jing uploaded a new patch addressed my comment and other problems on Aug 22. Jing uploaded another patch with minor change on Sep 6. I spent a few days reviewing and testing the patch and then I +1'ed and committed it on Sep 10. Quite a few people including you provided comments on the earlier patches but no additional comments were posted. I believed there was enough time for people to review it.
          Hide
          Eli Collins added a comment -

          Updating the fix version to 2.0.3 since it looks like Nicholas merged this?

          Show
          Eli Collins added a comment - Updating the fix version to 2.0.3 since it looks like Nicholas merged this?
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Eli, you are right that this was merged to branch-2. Thanks.

          Show
          Tsz Wo Nicholas Sze added a comment - Eli, you are right that this was merged to branch-2. Thanks.

            People

            • Assignee:
              Jing Zhao
              Reporter:
              Zhanwei Wang
            • Votes:
              0 Vote for this issue
              Watchers:
              28 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development