Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-826

Allow a mechanism for an application to detect that datanode(s) have died in the write pipeline

    Details

    • Hadoop Flags:
      Reviewed
    • Tags:
      hbase

      Description

      HDFS does not replicate the last block of the file that is being currently written to by an application. Every datanode death in the write pipeline decreases the reliability of the last block of the currently-being-written block. This situation can be improved if the application can be notified of a datanode death in the write pipeline. Then, the application can decide what is the right course of action to be taken on this event.

      In our use-case, the application can close the file on the first datanode death, and start writing to a newly created file. This ensures that the reliability guarantee of a block is close to 3 at all time.

      One idea is to make DFSOutoutStream. write() throw an exception if the number of datanodes in the write pipeline fall below minimum.replication.factor that is set on the client (this is backward compatible).

      1. HDFS-826.20-security.1.patch
        5 kB
        Jitendra Nath Pandey
      2. HDFS-826-0.20.patch
        7 kB
        Nicolas Spiegelberg
      3. HDFS-826-0.20-v2.patch
        5 kB
        stack
      4. Replicable4.txt
        5 kB
        dhruba borthakur
      5. ReplicableHdfs.txt
        3 kB
        dhruba borthakur
      6. ReplicableHdfs2.txt
        5 kB
        dhruba borthakur
      7. ReplicableHdfs3.txt
        5 kB
        dhruba borthakur

        Issue Links

          Activity

          Hide
          Hairong Kuang added a comment -

          The problem is that if datanode# in the write pipeline fall below minimum.replication.factor, a file is not able to be closed.

          Show
          Hairong Kuang added a comment - The problem is that if datanode# in the write pipeline fall below minimum.replication.factor, a file is not able to be closed.
          Hide
          dhruba borthakur added a comment -

          The application is such that it can start writing to a new file as soon as the one datanode in the write pipeline failed. Is there an alternate way to achieve this behaviour?

          Show
          dhruba borthakur added a comment - The application is such that it can start writing to a new file as soon as the one datanode in the write pipeline failed. Is there an alternate way to achieve this behaviour?
          Hide
          Hairong Kuang added a comment -

          Would polling be good enough for your application? DFSOutputStream provides an API to check if the number of active datanodes in the pipeline falls below the replication factor.

          The ultimate solution is not to decrease the length of the write pipeline on errors. This is especially important for the block that's reopen for append. I have been thinking about how to solve the problem and would like to put this to be on the top of my work items. I will open a jira once I have any presentable design.

          Show
          Hairong Kuang added a comment - Would polling be good enough for your application? DFSOutputStream provides an API to check if the number of active datanodes in the pipeline falls below the replication factor. The ultimate solution is not to decrease the length of the write pipeline on errors. This is especially important for the block that's reopen for append. I have been thinking about how to solve the problem and would like to put this to be on the top of my work items. I will open a jira once I have any presentable design.
          Hide
          dhruba borthakur added a comment -

          Actually, my original proposal might work with a twist. if we have a new configuration setting that says that a client should throw an error when the minimum replication factor drops from 2 to 1. In this case, the client will get an error from the write when the number of datanodes in the pipeline falls from 2 to 1. It would then invoke close() on the file which will success because the min.replication.factor is still set to 1 and there is one good datanode that has a valid copy of the block. The downside is that this is yet another new configuration parameter.

          Show
          dhruba borthakur added a comment - Actually, my original proposal might work with a twist. if we have a new configuration setting that says that a client should throw an error when the minimum replication factor drops from 2 to 1. In this case, the client will get an error from the write when the number of datanodes in the pipeline falls from 2 to 1. It would then invoke close() on the file which will success because the min.replication.factor is still set to 1 and there is one good datanode that has a valid copy of the block. The downside is that this is yet another new configuration parameter.
          Hide
          dhruba borthakur added a comment -

          > Would polling be good enough for your application?

          Polling might work, which api should I use? I am currently using FSDataoutputStream.write() to write new data to the file.

          Show
          dhruba borthakur added a comment - > Would polling be good enough for your application? Polling might work, which api should I use? I am currently using FSDataoutputStream.write() to write new data to the file.
          Hide
          Hairong Kuang added a comment -

          Unfortunately FSDataoutputStream needs a new API to support polling, either.

          Show
          Hairong Kuang added a comment - Unfortunately FSDataoutputStream needs a new API to support polling, either.
          Hide
          dhruba borthakur added a comment -

          Hi Hairong, would you support creating a new API in FSDataOutputStream to indicate the current number of replicas of the block at current offset?

          Show
          dhruba borthakur added a comment - Hi Hairong, would you support creating a new API in FSDataOutputStream to indicate the current number of replicas of the block at current offset?
          Hide
          dhruba borthakur added a comment -

          Returns the number of valid replicas of block that is currently being written into.

          Show
          dhruba borthakur added a comment - Returns the number of valid replicas of block that is currently being written into.
          Hide
          dhruba borthakur added a comment -

          One question that arises is what to return when the DFSClient has not yet allocated any blocks (or in-between new block allocations). I suspect we should return t intended replication factor of the file.

          Show
          dhruba borthakur added a comment - One question that arises is what to return when the DFSClient has not yet allocated any blocks (or in-between new block allocations). I suspect we should return t intended replication factor of the file.
          Hide
          Hairong Kuang added a comment -

          > I suspect we should return t intended replication factor of the file.
          This sounds good to me.

          Dhruba, is it possible that we do not introduce Replicable interface and do not propagate the change to FSOutputStream?

          Show
          Hairong Kuang added a comment - > I suspect we should return t intended replication factor of the file. This sounds good to me. Dhruba, is it possible that we do not introduce Replicable interface and do not propagate the change to FSOutputStream?
          Hide
          dhruba borthakur added a comment -

          If we do not introduce Replicable, then applications have to typecast to DFSOutputStream to use the new feature. So, essentially there is not much difference as far as maintaining this API as a true public facing API is concerned. Do you agree?

          Show
          dhruba borthakur added a comment - If we do not introduce Replicable, then applications have to typecast to DFSOutputStream to use the new feature. So, essentially there is not much difference as far as maintaining this API as a true public facing API is concerned. Do you agree?
          Hide
          Hairong Kuang added a comment -

          I am not strongly against introducing Replicable. But introducing the API to only DFSOutputStream has its advantage that the change is limited to DFS as what did with getting visible length.

          Show
          Hairong Kuang added a comment - I am not strongly against introducing Replicable. But introducing the API to only DFSOutputStream has its advantage that the change is limited to DFS as what did with getting visible length.
          Hide
          dhruba borthakur added a comment -

          My thinking is that most hadoop apps should be writing to the o.a.h.FileSystem or o.a.h.FileContext API (instead of the HDFS API). if we do that, then an app that runs well on HDFS is mostly likely to work on S3 or Ftp or whoever else implement o.a.h.FileContext API. Isn't that better than exposing many HDFS specific APIs?

          Show
          dhruba borthakur added a comment - My thinking is that most hadoop apps should be writing to the o.a.h.FileSystem or o.a.h.FileContext API (instead of the HDFS API). if we do that, then an app that runs well on HDFS is mostly likely to work on S3 or Ftp or whoever else implement o.a.h.FileContext API. Isn't that better than exposing many HDFS specific APIs?
          Hide
          Konstantin Shvachko added a comment -

          I remember we intended to implement but haven't done it yet two things related to this issue.

          1. The client throws an exception to the application when the write pipeline falls below the minimal replication factor.
          2. A client should be able to close a file even if its last block is not complete. With the following semantics: if the last block has at least one valid replica it will be fully replicated, otherwise the last block is treated as a corrupt block.

          It seems the patch proposes new api to work around the problems rather than addressing them directly.

          Show
          Konstantin Shvachko added a comment - I remember we intended to implement but haven't done it yet two things related to this issue. The client throws an exception to the application when the write pipeline falls below the minimal replication factor. A client should be able to close a file even if its last block is not complete. With the following semantics: if the last block has at least one valid replica it will be fully replicated, otherwise the last block is treated as a corrupt block. It seems the patch proposes new api to work around the problems rather than addressing them directly.
          Hide
          ryan rawson added a comment -

          Why wouldn't DFSClient just handle the write pipeline problems and just move on?

          Otherwise it sounds like if the client gets an error it must close the file, delete the file, then re-try again? This doesn't sound like a robust filesystem to me...

          Show
          ryan rawson added a comment - Why wouldn't DFSClient just handle the write pipeline problems and just move on? Otherwise it sounds like if the client gets an error it must close the file, delete the file, then re-try again? This doesn't sound like a robust filesystem to me...
          Hide
          dhruba borthakur added a comment -

          > If the client gets an error it must close the file, delete the file, then re-try again

          The idea is that the client should avoid getting an error. It can avoid getting an error if it polls the dfsfilehandle periodically and as soon as the number of active replicas decrease from 3 to 2, it can close the fie and start logging to a new file. The close will be successful because there are still 2 valid replicas of the block in HDFS.

          Show
          dhruba borthakur added a comment - > If the client gets an error it must close the file, delete the file, then re-try again The idea is that the client should avoid getting an error. It can avoid getting an error if it polls the dfsfilehandle periodically and as soon as the number of active replicas decrease from 3 to 2, it can close the fie and start logging to a new file. The close will be successful because there are still 2 valid replicas of the block in HDFS.
          Hide
          Eli Collins added a comment -

          To clarify by "client" you mean FileContext right? ie from the perspective of a client application that uses FileContext the underlying failures are resolved transparently if possible.

          Show
          Eli Collins added a comment - To clarify by "client" you mean FileContext right? ie from the perspective of a client application that uses FileContext the underlying failures are resolved transparently if possible.
          Hide
          dhruba borthakur added a comment -

          > To clarify by "client" you mean FileContext right? ie from the perspective of a client application that uses FileContext the underlying failures are resolved transparently if possible.

          Yes, that's right. I meant client == application

          Show
          dhruba borthakur added a comment - > To clarify by "client" you mean FileContext right? ie from the perspective of a client application that uses FileContext the underlying failures are resolved transparently if possible. Yes, that's right. I meant client == application
          Hide
          dhruba borthakur added a comment -

          I addressed Hairong's concern and kept this confined to DFSClient. I reverted back the proposed change to fs.FSDataOutputStream. I also added a unit test. Please let me know if this passes your review.

          Show
          dhruba borthakur added a comment - I addressed Hairong's concern and kept this confined to DFSClient. I reverted back the proposed change to fs.FSDataOutputStream. I also added a unit test. Please let me know if this passes your review.
          Hide
          dhruba borthakur added a comment -

          This patch is needed to make Hbase work elegantly on Hadoop 0.20.

          Show
          dhruba borthakur added a comment - This patch is needed to make Hbase work elegantly on Hadoop 0.20.
          Hide
          dhruba borthakur added a comment -

          Marged patch with latest trunk.

          Show
          dhruba borthakur added a comment - Marged patch with latest trunk.
          Hide
          dhruba borthakur added a comment -

          Can somebody pl review this patch? This is needed to make HBase work efficiently. Thanks.

          Show
          dhruba borthakur added a comment - Can somebody pl review this patch? This is needed to make HBase work efficiently. Thanks.
          Hide
          Todd Lipcon added a comment -

          Patch looks good to me. I question whether returning 0 for the case in between blocks is a good idea - this seems a bit confusing from the API user's perspective. Since it is well documented it may not be an issue, but I wonder if it would make more sense to actually return the intended replication in this case.

          Show
          Todd Lipcon added a comment - Patch looks good to me. I question whether returning 0 for the case in between blocks is a good idea - this seems a bit confusing from the API user's perspective. Since it is well documented it may not be an issue, but I wonder if it would make more sense to actually return the intended replication in this case.
          Hide
          dhruba borthakur added a comment -

          Thanks Todd for the review. I agree with your recommendation and will post a new patch.

          Show
          dhruba borthakur added a comment - Thanks Todd for the review. I agree with your recommendation and will post a new patch.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12436152/ReplicableHdfs3.txt
          against trunk revision 916902.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/255/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/255/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/255/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/255/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12436152/ReplicableHdfs3.txt against trunk revision 916902. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/255/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/255/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/255/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/255/console This message is automatically generated.
          Hide
          Konstantin Shvachko added a comment -

          What is the point of introducing new Replicable interface if it is not used anywhere? The new method getNumCurrentReplicas() in FSOutputSummer would work fine.

          Show
          Konstantin Shvachko added a comment - What is the point of introducing new Replicable interface if it is not used anywhere? The new method getNumCurrentReplicas() in FSOutputSummer would work fine.
          Hide
          stack added a comment -

          @ Konstantin It looks like DFSOS implements it:

          +class DFSOutputStream extends FSOutputSummer implements Syncable, Replicable {
          

          I'd have called the interface something other than Replicable though (Replicas?) since former seems like an attribute of interest to a client – as in the Stream is "replicable" so client somehow can call methods to replicate – rather than an attribute the stream is managing itself – as in "This stream uses or has replicas".

          Change suggested by Todd sounds good.

          Otherwise, patch looks good to me.

          Show
          stack added a comment - @ Konstantin It looks like DFSOS implements it: +class DFSOutputStream extends FSOutputSummer implements Syncable, Replicable { I'd have called the interface something other than Replicable though (Replicas?) since former seems like an attribute of interest to a client – as in the Stream is "replicable" so client somehow can call methods to replicate – rather than an attribute the stream is managing itself – as in "This stream uses or has replicas". Change suggested by Todd sounds good. Otherwise, patch looks good to me.
          Hide
          stack added a comment -

          If making a new patch, take a look at the javadoc for the method getNumCurrentReplicas. Add a @return at least to the interface.

          Show
          stack added a comment - If making a new patch, take a look at the javadoc for the method getNumCurrentReplicas. Add a @return at least to the interface.
          Hide
          Hairong Kuang added a comment -

          I do not think that Replicable interface is necessary.

          Show
          Hairong Kuang added a comment - I do not think that Replicable interface is necessary.
          Hide
          dhruba borthakur added a comment -

          Cancelling to incorporate review comments.

          Show
          dhruba borthakur added a comment - Cancelling to incorporate review comments.
          Hide
          dhruba borthakur added a comment -

          1. Removed replicable interface
          2. If it is in-between blocks and there isn't a valid pipleline, then return the replicationFactor of the file

          Show
          dhruba borthakur added a comment - 1. Removed replicable interface 2. If it is in-between blocks and there isn't a valid pipleline, then return the replicationFactor of the file
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12437700/Replicable4.txt
          against trunk revision 916902.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/124/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/124/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/124/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/124/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12437700/Replicable4.txt against trunk revision 916902. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/124/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/124/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/124/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/124/console This message is automatically generated.
          Hide
          dhruba borthakur added a comment -

          anybody wants to re-review this one? Thanks.

          Show
          dhruba borthakur added a comment - anybody wants to re-review this one? Thanks.
          Hide
          stack added a comment -

          Patch looks good to me (Didn't try it, just reviewed patch).

          Show
          stack added a comment - Patch looks good to me (Didn't try it, just reviewed patch).
          Hide
          Hairong Kuang added a comment -

          Dhruba, sorry for this late thought. It might be nicer to return -1 when pipeline has not been set up yet. This simplifies code and provides more information to the user because it can distinguish if a pipeline has no error from it is not up yet.

          Show
          Hairong Kuang added a comment - Dhruba, sorry for this late thought. It might be nicer to return -1 when pipeline has not been set up yet. This simplifies code and provides more information to the user because it can distinguish if a pipeline has no error from it is not up yet.
          Hide
          dhruba borthakur added a comment -

          Hauirong: it makes the application code easier when the getNumCurrentReplicas() return the expected number of replicas when there isn't a pipeline. This also makes the API somewhat opaque to pipeline-setup internals.

          Show
          dhruba borthakur added a comment - Hauirong: it makes the application code easier when the getNumCurrentReplicas() return the expected number of replicas when there isn't a pipeline. This also makes the API somewhat opaque to pipeline-setup internals.
          Hide
          Hairong Kuang added a comment -

          I do not think that making the user aware of whether the replicas are created or not is a bad idea. Why hiding this is better? Less code change but providing the user more information seems to me is very good. But I do not mean that you have to do it this way. It is just a suggestion.

          Show
          Hairong Kuang added a comment - I do not think that making the user aware of whether the replicas are created or not is a bad idea. Why hiding this is better? Less code change but providing the user more information seems to me is very good. But I do not mean that you have to do it this way. It is just a suggestion.
          Hide
          dhruba borthakur added a comment -

          Hi Hairong, Thanks for the review. I would really like to continue with the approach of making getNumCurrentReplicas return the default replication factor when there isn't a pipeline. One way that I am thinking is that the app can visualize one file to be one huge ever-expanding block and as long as there are no pipeline errors, the app will always see a NumReplicas as the default replication factor. I checked this with a few HBase folks and it appears simpler to them from an application programmer's perspective.

          I hope this is ok with you and thanks for reviewing it.

          Show
          dhruba borthakur added a comment - Hi Hairong, Thanks for the review. I would really like to continue with the approach of making getNumCurrentReplicas return the default replication factor when there isn't a pipeline. One way that I am thinking is that the app can visualize one file to be one huge ever-expanding block and as long as there are no pipeline errors, the app will always see a NumReplicas as the default replication factor. I checked this with a few HBase folks and it appears simpler to them from an application programmer's perspective. I hope this is ok with you and thanks for reviewing it.
          Hide
          stack added a comment -

          Trying to make a version of this patch for 0.20 and having a bit of trouble. In particular, DataStreamer#getNodes doesn't seem easy to do in 0.20 context... DataStreamer is very different.

          Show
          stack added a comment - Trying to make a version of this patch for 0.20 and having a bit of trouble. In particular, DataStreamer#getNodes doesn't seem easy to do in 0.20 context... DataStreamer is very different.
          Hide
          Nicolas Spiegelberg added a comment -

          Attached a patch that is compatible with 0.20

          Show
          Nicolas Spiegelberg added a comment - Attached a patch that is compatible with 0.20
          Hide
          stack added a comment -

          A version of Nicolas' patch absent minus the Replicable interface.

          Show
          stack added a comment - A version of Nicolas' patch absent minus the Replicable interface.
          Hide
          dhruba borthakur added a comment -

          Thanks stack for reviewing this one. I will commit it.

          Show
          dhruba borthakur added a comment - Thanks stack for reviewing this one. I will commit it.
          Hide
          Hairong Kuang added a comment -

          > hope this is ok with you and thanks for reviewing it.
          As I said, I am not against your approach. Feel free to commit it if you feel strongly about it.

          Show
          Hairong Kuang added a comment - > hope this is ok with you and thanks for reviewing it. As I said, I am not against your approach. Feel free to commit it if you feel strongly about it.
          Hide
          stack added a comment -

          Just to say that I'm +1 on committing Replicable4.txt (I've already added the 0.20 version, HDFS-826-0.20-v2.patch, of this patch to the hadoop we bundle with hbase).

          Show
          stack added a comment - Just to say that I'm +1 on committing Replicable4.txt (I've already added the 0.20 version, HDFS-826 -0.20-v2.patch, of this patch to the hadoop we bundle with hbase).
          Hide
          dhruba borthakur added a comment -

          I just committed this.

          Show
          dhruba borthakur added a comment - I just committed this.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #214 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/214/)
          . The DFSOutputStream has a API that returns the number of
          active datanode(s) in the current pipeline. (dhruba)

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #214 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/214/ ) . The DFSOutputStream has a API that returns the number of active datanode(s) in the current pipeline. (dhruba)
          Hide
          Hudson added a comment -

          Integrated in Hdfs-Patch-h2.grid.sp2.yahoo.net #146 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/146/)

          Show
          Hudson added a comment - Integrated in Hdfs-Patch-h2.grid.sp2.yahoo.net #146 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/146/ )
          Hide
          Hudson added a comment -

          Integrated in Hdfs-Patch-h5.grid.sp2.yahoo.net #302 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/302/)

          Show
          Hudson added a comment - Integrated in Hdfs-Patch-h5.grid.sp2.yahoo.net #302 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/302/ )
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #275 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/275/)

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #275 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/275/ )
          Hide
          Nicolas Spiegelberg added a comment -

          This should be pulled into the branch-0.20-append branch.

          Show
          Nicolas Spiegelberg added a comment - This should be pulled into the branch-0.20-append branch.
          Hide
          Nicolas Spiegelberg added a comment -

          Note: HDFS-826-0.20-v2.patch can be applied to the 0.20-append branch as is.

          Show
          Nicolas Spiegelberg added a comment - Note: HDFS-826 -0.20-v2.patch can be applied to the 0.20-append branch as is.
          Hide
          Jitendra Nath Pandey added a comment -

          Uploaded patch for 20-security branch.

          Show
          Jitendra Nath Pandey added a comment - Uploaded patch for 20-security branch.
          Hide
          Suresh Srinivas added a comment -

          +1 for the patch. I committed the patch to 0.20-security branch.

          Show
          Suresh Srinivas added a comment - +1 for the patch. I committed the patch to 0.20-security branch.

            People

            • Assignee:
              dhruba borthakur
              Reporter:
              dhruba borthakur
            • Votes:
              0 Vote for this issue
              Watchers:
              16 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development