Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-234

Integration with BookKeeper logging system

    Details

    • Type: New Feature New Feature
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.0.0-alpha, 3.0.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      BookKeeper is a system to reliably log streams of records (https://issues.apache.org/jira/browse/ZOOKEEPER-276). The NameNode is a natural target for such a system for being the metadata repository of the entire file system for HDFS.

      1. create.png
        3 kB
        Luca Telloli
      2. HADOOP-5189-trunk-preview.patch
        33 kB
        Luca Telloli
      3. HADOOP-5189-trunk-preview.patch
        54 kB
        Luca Telloli
      4. HADOOP-5189-trunk-preview.patch
        55 kB
        Flavio Junqueira
      5. HADOOP-5189.patch
        58 kB
        Luca Telloli
      6. HADOOP-5189-v.19.patch
        30 kB
        Luca Telloli
      7. HDFS-234.patch
        33 kB
        Luca Telloli
      8. zookeeper-dev.jar
        887 kB
        Luca Telloli
      9. zookeeper-dev-bookkeeper.jar
        125 kB
        Luca Telloli
      10. HDFS-234.diff
        63 kB
        Ivan Kelly
      11. hdfs_tpt_lat.pdf
        20 kB
        Ivan Kelly
      12. HDFS-234.diff
        63 kB
        Ivan Kelly
      13. HDFS-234.diff
        63 kB
        Ivan Kelly
      14. HDFS-234.diff
        63 kB
        Ivan Kelly
      15. HDFS-234.diff
        71 kB
        Ivan Kelly
      16. HDFS-234-branch-2.patch
        73 kB
        Ivan Kelly

        Issue Links

          Activity

          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1207 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1207/)
          Moving HDFS-234 from trunk to Release 2.0.0-alpha section (Revision 1389494)

          Result = SUCCESS
          suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1389494
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1207 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1207/ ) Moving HDFS-234 from trunk to Release 2.0.0-alpha section (Revision 1389494) Result = SUCCESS suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1389494 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1176 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1176/)
          Moving HDFS-234 from trunk to Release 2.0.0-alpha section (Revision 1389494)

          Result = SUCCESS
          suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1389494
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1176 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1176/ ) Moving HDFS-234 from trunk to Release 2.0.0-alpha section (Revision 1389494) Result = SUCCESS suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1389494 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #2783 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2783/)
          Moving HDFS-234 from trunk to Release 2.0.0-alpha section (Revision 1389494)

          Result = FAILURE
          suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1389494
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2783 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2783/ ) Moving HDFS-234 from trunk to Release 2.0.0-alpha section (Revision 1389494) Result = FAILURE suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1389494 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #2824 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2824/)
          Moving HDFS-234 from trunk to Release 2.0.0-alpha section (Revision 1389494)

          Result = SUCCESS
          suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1389494
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2824 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2824/ ) Moving HDFS-234 from trunk to Release 2.0.0-alpha section (Revision 1389494) Result = SUCCESS suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1389494 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #2761 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2761/)
          Moving HDFS-234 from trunk to Release 2.0.0-alpha section (Revision 1389494)

          Result = SUCCESS
          suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1389494
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2761 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2761/ ) Moving HDFS-234 from trunk to Release 2.0.0-alpha section (Revision 1389494) Result = SUCCESS suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1389494 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          Uma Maheswara Rao G added a comment -

          I have just pushed this to Hadoop-2 and alpha cut. Thanks a lot Ivan.

          Show
          Uma Maheswara Rao G added a comment - I have just pushed this to Hadoop-2 and alpha cut. Thanks a lot Ivan.
          Hide
          Nathan Roberts added a comment -

          I agree it would be good to have it in 2.0-alpha for folks to experiment with and compare with other alternatives.

          Show
          Nathan Roberts added a comment - I agree it would be good to have it in 2.0-alpha for folks to experiment with and compare with other alternatives.
          Hide
          Uma Maheswara Rao G added a comment -

          Others, Agreeing with Suresh?

          Show
          Uma Maheswara Rao G added a comment - Others, Agreeing with Suresh?
          Hide
          Uma Maheswara Rao G added a comment -

          Eli, HDFS-2717 and HDFS-3058 are the main issues identified at BKJM side with testing and discussions with Ivan. Also we have been filing some in BK project also.

          given that is this an option that Uma is trying out and 2.0-alpha is being release, I feel it is fine for HDFS-234 to be merged to 2.0.

          Yes, Suresh, this is what my opinion.

          Show
          Uma Maheswara Rao G added a comment - Eli, HDFS-2717 and HDFS-3058 are the main issues identified at BKJM side with testing and discussions with Ivan. Also we have been filing some in BK project also. given that is this an option that Uma is trying out and 2.0-alpha is being release, I feel it is fine for HDFS-234 to be merged to 2.0. Yes, Suresh, this is what my opinion.
          Hide
          Suresh Srinivas added a comment -

          Eli, given that is this an option that Uma is trying out and 2.0-alpha is being release, I feel it is fine for HDFS-234 to be merged to 2.0. As regards to contrib components, lets have a separate discussion on how BK related code should be made available, if not contrib.

          Show
          Suresh Srinivas added a comment - Eli, given that is this an option that Uma is trying out and 2.0-alpha is being release, I feel it is fine for HDFS-234 to be merged to 2.0. As regards to contrib components, lets have a separate discussion on how BK related code should be made available, if not contrib.
          Hide
          Eli Collins added a comment -

          Uma,

          Given all the bugs you found recently, and haven't been triaged/assigned, there are no docs, etc I'd keep this slated for 3.0. If it gets tested/doc'd and stabilizes on trunk then we can consider merging.

          We're actually trying to get away from using contrib. See this thread for history: http://search-hadoop.com/m/EPdTjoLpuh2. I removed most of the hdfs contrib components.

          Thanks,
          Eli

          Show
          Eli Collins added a comment - Uma, Given all the bugs you found recently, and haven't been triaged/assigned, there are no docs, etc I'd keep this slated for 3.0. If it gets tested/doc'd and stabilizes on trunk then we can consider merging. We're actually trying to get away from using contrib. See this thread for history: http://search-hadoop.com/m/EPdTjoLpuh2 . I removed most of the hdfs contrib components. Thanks, Eli
          Hide
          dhruba borthakur added a comment -

          Hi Uma, while it is true that we are dedicating resources to evaluate HDFS-HA with BK, it will possibly be for trunk and not for 2.0. are you folks planning to use this in hadoop 2.0 in production?

          Show
          dhruba borthakur added a comment - Hi Uma, while it is true that we are dedicating resources to evaluate HDFS-HA with BK, it will possibly be for trunk and not for 2.0. are you folks planning to use this in hadoop 2.0 in production?
          Hide
          Uma Maheswara Rao G added a comment -

          Hi Todd,
          As this is also one option for HA, we are working on this patch merged to Hadoop-2 internally and we are testing with BookKeeper. Also this is a contrib part in HDFS. As this is a pluggable option, why can't we keep this in contrib? If we have other options ready and performing well, then we can choose one among, as you mentioned in other discussions.

          I have seen the comment from Dhruba for continue invest on BK based approach also. https://issues.apache.org/jira/browse/HDFS-3092?focusedCommentId=13265110&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13265110
          Since the basic thing ready from BKJM side, it will help some users for evaluating it as this is an alpha quality release. Identified some issues like HDFS-3058 in our testing also. Infact we have to push this also in.

          Regarding document: I agree, we may have to prepare well documentation for using this option, currently it is available only in BKJM class.

          Show
          Uma Maheswara Rao G added a comment - Hi Todd, As this is also one option for HA, we are working on this patch merged to Hadoop-2 internally and we are testing with BookKeeper. Also this is a contrib part in HDFS. As this is a pluggable option, why can't we keep this in contrib? If we have other options ready and performing well, then we can choose one among, as you mentioned in other discussions. I have seen the comment from Dhruba for continue invest on BK based approach also. https://issues.apache.org/jira/browse/HDFS-3092?focusedCommentId=13265110&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13265110 Since the basic thing ready from BKJM side, it will help some users for evaluating it as this is an alpha quality release. Identified some issues like HDFS-3058 in our testing also. Infact we have to push this also in. Regarding document: I agree, we may have to prepare well documentation for using this option, currently it is available only in BKJM class.
          Hide
          Todd Lipcon added a comment -

          Uma: I'm a little concerned about having this as a main part of our 2.0 release, to be honest. There hasn't been activity and the only activity I've seen around it has been from Ivan. Is anyone committed to actually QAing and running this code? A lot of the other features around HA don't integrate with it, it isn't documented, etc. Maybe we should start a discussion on the list.

          Show
          Todd Lipcon added a comment - Uma: I'm a little concerned about having this as a main part of our 2.0 release, to be honest. There hasn't been activity and the only activity I've seen around it has been from Ivan. Is anyone committed to actually QAing and running this code? A lot of the other features around HA don't integrate with it, it isn't documented, etc. Maybe we should start a discussion on the list.
          Hide
          Ivan Kelly added a comment -

          This patch is a direct copy of the bkjournal code from trunk on the 8th May 2012. To be applied on branch-2.

          Show
          Ivan Kelly added a comment - This patch is a direct copy of the bkjournal code from trunk on the 8th May 2012. To be applied on branch-2.
          Hide
          Uma Maheswara Rao G added a comment -

          Looks this particular issue, did not merged to HA branch ( but marked fix versions to HA branch also).
          It was committed only to trunk. So, 23 branch missing this code.

          We have to merge this 23 branch for HA release right.( as this is also one option in shared storage).?

          @Aaron, are we planning this issue merge to 23?

          Show
          Uma Maheswara Rao G added a comment - Looks this particular issue, did not merged to HA branch ( but marked fix versions to HA branch also). It was committed only to trunk. So, 23 branch missing this code. We have to merge this 23 branch for HA release right.( as this is also one option in shared storage).? @Aaron, are we planning this issue merge to 23?
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #933 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/933/)
          Amend HDFS-234 (bkjournal contrib). Fix findbugs build

          • Fixes pom parent to point to hadoop-project instead of hadoop-project-dist
          • Fix relative path in pom to have the correct number of ../s

          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1220940
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/pom.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #933 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/933/ ) Amend HDFS-234 (bkjournal contrib). Fix findbugs build Fixes pom parent to point to hadoop-project instead of hadoop-project-dist Fix relative path in pom to have the correct number of ../s todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1220940 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/pom.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #900 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/900/)
          Amend HDFS-234 (bkjournal contrib). Fix findbugs build

          • Fixes pom parent to point to hadoop-project instead of hadoop-project-dist
          • Fix relative path in pom to have the correct number of ../s

          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1220940
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/pom.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #900 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/900/ ) Amend HDFS-234 (bkjournal contrib). Fix findbugs build Fixes pom parent to point to hadoop-project instead of hadoop-project-dist Fix relative path in pom to have the correct number of ../s todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1220940 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/pom.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #1476 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1476/)
          Amend HDFS-234 (bkjournal contrib). Fix findbugs build

          • Fixes pom parent to point to hadoop-project instead of hadoop-project-dist
          • Fix relative path in pom to have the correct number of ../s

          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1220940
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/pom.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #1476 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1476/ ) Amend HDFS-234 (bkjournal contrib). Fix findbugs build Fixes pom parent to point to hadoop-project instead of hadoop-project-dist Fix relative path in pom to have the correct number of ../s todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1220940 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/pom.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #1526 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1526/)
          Amend HDFS-234 (bkjournal contrib). Fix findbugs build

          • Fixes pom parent to point to hadoop-project instead of hadoop-project-dist
          • Fix relative path in pom to have the correct number of ../s

          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1220940
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/pom.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #1526 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1526/ ) Amend HDFS-234 (bkjournal contrib). Fix findbugs build Fixes pom parent to point to hadoop-project instead of hadoop-project-dist Fix relative path in pom to have the correct number of ../s todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1220940 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/pom.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #1453 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1453/)
          Amend HDFS-234 (bkjournal contrib). Fix findbugs build

          • Fixes pom parent to point to hadoop-project instead of hadoop-project-dist
          • Fix relative path in pom to have the correct number of ../s

          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1220940
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/pom.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #1453 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1453/ ) Amend HDFS-234 (bkjournal contrib). Fix findbugs build Fixes pom parent to point to hadoop-project instead of hadoop-project-dist Fix relative path in pom to have the correct number of ../s todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1220940 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/pom.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #894 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/894/)
          HDFS-234. Integration with BookKeeper logging system. Contributed by Ivan Kelly.

          jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1213983
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/README.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/pom.xml
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/BookKeeperEditLogInputStream.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/BookKeeperEditLogOutputStream.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/BookKeeperJournalManager.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/EditLogLedgerMetadata.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/MaxTxId.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/WriteLock.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal/TestBookKeeperJournalManager.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/hdfs
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/hdfs/server
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/hdfs/server/namenode
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogTestUtil.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/resources
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/resources/log4j.properties
          • /hadoop/common/trunk/hadoop-hdfs-project/pom.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #894 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/894/ ) HDFS-234 . Integration with BookKeeper logging system. Contributed by Ivan Kelly. jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1213983 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/README.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/pom.xml /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/BookKeeperEditLogInputStream.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/BookKeeperEditLogOutputStream.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/BookKeeperJournalManager.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/EditLogLedgerMetadata.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/MaxTxId.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/WriteLock.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal/TestBookKeeperJournalManager.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/hdfs /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/hdfs/server /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/hdfs/server/namenode /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogTestUtil.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/resources /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/resources/log4j.properties /hadoop/common/trunk/hadoop-hdfs-project/pom.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #927 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/927/)
          HDFS-234. Integration with BookKeeper logging system. Contributed by Ivan Kelly.

          jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1213983
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/README.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/pom.xml
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/BookKeeperEditLogInputStream.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/BookKeeperEditLogOutputStream.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/BookKeeperJournalManager.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/EditLogLedgerMetadata.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/MaxTxId.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/WriteLock.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal/TestBookKeeperJournalManager.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/hdfs
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/hdfs/server
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/hdfs/server/namenode
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogTestUtil.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/resources
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/resources/log4j.properties
          • /hadoop/common/trunk/hadoop-hdfs-project/pom.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #927 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/927/ ) HDFS-234 . Integration with BookKeeper logging system. Contributed by Ivan Kelly. jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1213983 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/README.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/pom.xml /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/BookKeeperEditLogInputStream.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/BookKeeperEditLogOutputStream.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/BookKeeperJournalManager.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/EditLogLedgerMetadata.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/MaxTxId.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/WriteLock.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal/TestBookKeeperJournalManager.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/hdfs /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/hdfs/server /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/hdfs/server/namenode /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogTestUtil.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/resources /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/resources/log4j.properties /hadoop/common/trunk/hadoop-hdfs-project/pom.xml
          Hide
          Ivan Kelly added a comment -

          Awesome Thanks for your help Jitendra.

          Show
          Ivan Kelly added a comment - Awesome Thanks for your help Jitendra.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #1443 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1443/)
          HDFS-234. Integration with BookKeeper logging system. Contributed by Ivan Kelly.

          jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1213983
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/README.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/pom.xml
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/BookKeeperEditLogInputStream.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/BookKeeperEditLogOutputStream.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/BookKeeperJournalManager.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/EditLogLedgerMetadata.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/MaxTxId.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/WriteLock.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal/TestBookKeeperJournalManager.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/hdfs
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/hdfs/server
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/hdfs/server/namenode
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogTestUtil.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/resources
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/resources/log4j.properties
          • /hadoop/common/trunk/hadoop-hdfs-project/pom.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #1443 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1443/ ) HDFS-234 . Integration with BookKeeper logging system. Contributed by Ivan Kelly. jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1213983 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/README.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/pom.xml /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/BookKeeperEditLogInputStream.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/BookKeeperEditLogOutputStream.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/BookKeeperJournalManager.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/EditLogLedgerMetadata.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/MaxTxId.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/WriteLock.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal/TestBookKeeperJournalManager.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/hdfs /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/hdfs/server /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/hdfs/server/namenode /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogTestUtil.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/resources /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/resources/log4j.properties /hadoop/common/trunk/hadoop-hdfs-project/pom.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #1419 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1419/)
          HDFS-234. Integration with BookKeeper logging system. Contributed by Ivan Kelly.

          jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1213983
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/README.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/pom.xml
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/BookKeeperEditLogInputStream.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/BookKeeperEditLogOutputStream.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/BookKeeperJournalManager.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/EditLogLedgerMetadata.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/MaxTxId.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/WriteLock.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal/TestBookKeeperJournalManager.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/hdfs
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/hdfs/server
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/hdfs/server/namenode
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogTestUtil.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/resources
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/resources/log4j.properties
          • /hadoop/common/trunk/hadoop-hdfs-project/pom.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #1419 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1419/ ) HDFS-234 . Integration with BookKeeper logging system. Contributed by Ivan Kelly. jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1213983 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/README.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/pom.xml /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/BookKeeperEditLogInputStream.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/BookKeeperEditLogOutputStream.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/BookKeeperJournalManager.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/EditLogLedgerMetadata.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/MaxTxId.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/WriteLock.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal/TestBookKeeperJournalManager.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/hdfs /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/hdfs/server /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/hdfs/server/namenode /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogTestUtil.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/resources /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/resources/log4j.properties /hadoop/common/trunk/hadoop-hdfs-project/pom.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #1493 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1493/)
          HDFS-234. Integration with BookKeeper logging system. Contributed by Ivan Kelly.

          jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1213983
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/README.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/pom.xml
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/BookKeeperEditLogInputStream.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/BookKeeperEditLogOutputStream.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/BookKeeperJournalManager.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/EditLogLedgerMetadata.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/MaxTxId.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/WriteLock.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal/TestBookKeeperJournalManager.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/hdfs
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/hdfs/server
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/hdfs/server/namenode
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogTestUtil.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/resources
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/resources/log4j.properties
          • /hadoop/common/trunk/hadoop-hdfs-project/pom.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #1493 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1493/ ) HDFS-234 . Integration with BookKeeper logging system. Contributed by Ivan Kelly. jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1213983 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/README.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/pom.xml /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/BookKeeperEditLogInputStream.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/BookKeeperEditLogOutputStream.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/BookKeeperJournalManager.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/EditLogLedgerMetadata.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/MaxTxId.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/WriteLock.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal/TestBookKeeperJournalManager.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/hdfs /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/hdfs/server /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/hdfs/server/namenode /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogTestUtil.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/resources /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/resources/log4j.properties /hadoop/common/trunk/hadoop-hdfs-project/pom.xml
          Hide
          Jitendra Nath Pandey added a comment -

          Committed to trunk. Thanks to Ivan Kelly for the great work!

          Show
          Jitendra Nath Pandey added a comment - Committed to trunk. Thanks to Ivan Kelly for the great work!
          Hide
          Jitendra Nath Pandey added a comment -

          +1 for the patch.

          Show
          Jitendra Nath Pandey added a comment - +1 for the patch.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12507213/HDFS-234.diff
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 6 new or modified tests.

          -1 javadoc. The javadoc tool appears to have generated 90 warning messages.

          -1 javac. The applied patch generated 39 javac compiler warnings (more than the trunk's current 37 warnings).

          -1 eclipse:eclipse. The patch failed to build with eclipse:eclipse.

          -1 findbugs. The patch appears to cause Findbugs (version 1.3.9) to fail.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hdfs.TestFileAppend
          org.apache.hadoop.hdfs.TestDFSRemove

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1704//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1704//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12507213/HDFS-234.diff against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. -1 javadoc. The javadoc tool appears to have generated 90 warning messages. -1 javac. The applied patch generated 39 javac compiler warnings (more than the trunk's current 37 warnings). -1 eclipse:eclipse. The patch failed to build with eclipse:eclipse. -1 findbugs. The patch appears to cause Findbugs (version 1.3.9) to fail. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hdfs.TestFileAppend org.apache.hadoop.hdfs.TestDFSRemove +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1704//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1704//console This message is automatically generated.
          Hide
          Ivan Kelly added a comment -

          Moved to contrib. Added README for running a namenode with BookKeeper.

          Show
          Ivan Kelly added a comment - Moved to contrib. Added README for running a namenode with BookKeeper.
          Hide
          Jakob Homan added a comment -

          I think we should commit it to hdfs contrib. Please move the bkjournal package to contrib.

          +1

          Show
          Jakob Homan added a comment - I think we should commit it to hdfs contrib. Please move the bkjournal package to contrib. +1
          Hide
          Ivan Kelly added a comment -

          > I think we should commit it to hdfs contrib. Please move the bkjournal package to contrib.
          In principle I have no problem with this. However, contrib doesn't seem to be set up to run with maven at the moment. This needs to be compiled and tested as part of the normal build cycle. Otherwise, changes to the NN interfaces will silently break this.

          Show
          Ivan Kelly added a comment - > I think we should commit it to hdfs contrib. Please move the bkjournal package to contrib. In principle I have no problem with this. However, contrib doesn't seem to be set up to run with maven at the moment. This needs to be compiled and tested as part of the normal build cycle. Otherwise, changes to the NN interfaces will silently break this.
          Hide
          Jitendra Nath Pandey added a comment -

          I think we should commit it to hdfs contrib. Please move the bkjournal package to contrib.

          For secure hadoop setup, it will be important to have authenticated connection to book keeper servers. That should be addressed in follow up jiras.

          Show
          Jitendra Nath Pandey added a comment - I think we should commit it to hdfs contrib. Please move the bkjournal package to contrib. For secure hadoop setup, it will be important to have authenticated connection to book keeper servers. That should be addressed in follow up jiras.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12506535/HDFS-234.diff
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          -1 javac. The applied patch generated 37 javac compiler warnings (more than the trunk's current 35 warnings).

          -1 findbugs. The patch appears to introduce 9 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in .

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1659//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1659//artifact/trunk/hadoop-hdfs-project/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1659//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12506535/HDFS-234.diff against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 4 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 37 javac compiler warnings (more than the trunk's current 35 warnings). -1 findbugs. The patch appears to introduce 9 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1659//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1659//artifact/trunk/hadoop-hdfs-project/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1659//console This message is automatically generated.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12506511/HDFS-234.diff
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          -1 javac. The patch appears to cause tar ant target to fail.

          -1 findbugs. The patch appears to cause Findbugs (version 1.3.9) to fail.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed the unit tests build

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1658//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1658//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12506511/HDFS-234.diff against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 4 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The patch appears to cause tar ant target to fail. -1 findbugs. The patch appears to cause Findbugs (version 1.3.9) to fail. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed the unit tests build +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1658//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1658//console This message is automatically generated.
          Hide
          Ivan Kelly added a comment -

          Changed to patch to make it use the official BK 4.0.0 build.

          Show
          Ivan Kelly added a comment - Changed to patch to make it use the official BK 4.0.0 build.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12506247/HDFS-234.diff
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          -1 javac. The patch appears to cause tar ant target to fail.

          -1 findbugs. The patch appears to cause Findbugs (version 1.3.9) to fail.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed the unit tests build

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1646//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1646//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12506247/HDFS-234.diff against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 4 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The patch appears to cause tar ant target to fail. -1 findbugs. The patch appears to cause Findbugs (version 1.3.9) to fail. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed the unit tests build +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1646//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1646//console This message is automatically generated.
          Hide
          Ivan Kelly added a comment -

          Reuploading patch, so that it runs through QA.

          Show
          Ivan Kelly added a comment - Reuploading patch, so that it runs through QA.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12503763/hdfs_tpt_lat.pdf
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1560//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12503763/hdfs_tpt_lat.pdf against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1560//console This message is automatically generated.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12503761/HDFS-234.diff
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          -1 javac. The patch appears to cause tar ant target to fail.

          -1 findbugs. The patch appears to cause Findbugs (version 1.3.9) to fail.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed the unit tests build

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1559//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1559//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12503761/HDFS-234.diff against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 4 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The patch appears to cause tar ant target to fail. -1 findbugs. The patch appears to cause Findbugs (version 1.3.9) to fail. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed the unit tests build +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1559//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1559//console This message is automatically generated.
          Hide
          Ivan Kelly added a comment -

          Added latency/tpt diagram.

          Show
          Ivan Kelly added a comment - Added latency/tpt diagram.
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/2835/
          -----------------------------------------------------------

          Review request for hadoop-hdfs.

          Summary
          -------

          BookKeeper is a system to reliably log streams of records (https://issues.apache.org/jira/browse/ZOOKEEPER-276). The NameNode is a natural target for such a system for being the metadata repository of the entire file system for HDFS.

          This addresses bug HDFS-234.
          http://issues.apache.org/jira/browse/HDFS-234

          Diffs


          hadoop-hdfs-project/hadoop-hdfs/pom.xml 1bcc372
          hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/bkjournal/BookKeeperEditLogInputStream.java PRE-CREATION
          hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/bkjournal/TestBookKeeperJournalManager.java PRE-CREATION
          hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogTestUtil.java PRE-CREATION
          hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/bkjournal/EditLogLedgerMetadata.java PRE-CREATION
          hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/bkjournal/MaxTxId.java PRE-CREATION
          hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/bkjournal/WriteLock.java PRE-CREATION
          hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/bkjournal/BookKeeperEditLogOutputStream.java PRE-CREATION
          hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/bkjournal/BookKeeperJournalManager.java PRE-CREATION

          Diff: https://reviews.apache.org/r/2835/diff

          Testing
          -------

          Thanks,

          Ivan

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2835/ ----------------------------------------------------------- Review request for hadoop-hdfs. Summary ------- BookKeeper is a system to reliably log streams of records ( https://issues.apache.org/jira/browse/ZOOKEEPER-276 ). The NameNode is a natural target for such a system for being the metadata repository of the entire file system for HDFS. This addresses bug HDFS-234 . http://issues.apache.org/jira/browse/HDFS-234 Diffs hadoop-hdfs-project/hadoop-hdfs/pom.xml 1bcc372 hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/bkjournal/BookKeeperEditLogInputStream.java PRE-CREATION hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/bkjournal/TestBookKeeperJournalManager.java PRE-CREATION hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogTestUtil.java PRE-CREATION hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/bkjournal/EditLogLedgerMetadata.java PRE-CREATION hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/bkjournal/MaxTxId.java PRE-CREATION hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/bkjournal/WriteLock.java PRE-CREATION hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/bkjournal/BookKeeperEditLogOutputStream.java PRE-CREATION hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/bkjournal/BookKeeperJournalManager.java PRE-CREATION Diff: https://reviews.apache.org/r/2835/diff Testing ------- Thanks, Ivan
          Hide
          Ivan Kelly added a comment -

          This patch depends on HDFS-1580

          Show
          Ivan Kelly added a comment - This patch depends on HDFS-1580
          Hide
          Ivan Kelly added a comment -

          Reviving this JIRA as the namenode journalling refactor is almost complete.

          This JIRA blocked by HDFS-2188, HDFS-2334 & HDFS-1580. Once these are in trunk I'll be able to cut a patch for this. Until then, I've got a beta of HDFS+PluggableJournal+BK on github.
          https://github.com/ivankelly/hadoop-common/tree/BKJournal

          Show
          Ivan Kelly added a comment - Reviving this JIRA as the namenode journalling refactor is almost complete. This JIRA blocked by HDFS-2188 , HDFS-2334 & HDFS-1580 . Once these are in trunk I'll be able to cut a patch for this. Until then, I've got a beta of HDFS+PluggableJournal+BK on github. https://github.com/ivankelly/hadoop-common/tree/BKJournal
          Hide
          Flavio Junqueira added a comment -

          When you read from a BookKeeper ledger, it seems that you are reading the whole ledger before processing it. I can see two potential problems with it:

          1. If the ledger is large, then the process reading the ledger might end up consuming too much memory;
          2. It does not overlap reading the ledger with processing. It might be best to try to overlap reading from the ledger with processing the edits. I don't know what changes this suggestion implies, though.
          Show
          Flavio Junqueira added a comment - When you read from a BookKeeper ledger, it seems that you are reading the whole ledger before processing it. I can see two potential problems with it: If the ledger is large, then the process reading the ledger might end up consuming too much memory; It does not overlap reading the ledger with processing. It might be best to try to overlap reading from the ledger with processing the edits. I don't know what changes this suggestion implies, though.
          Hide
          Luca Telloli added a comment -

          Zookeeper and Bookkeeper jars, until we sort out how to include them through ivy

          Show
          Luca Telloli added a comment - Zookeeper and Bookkeeper jars, until we sort out how to include them through ivy
          Hide
          Luca Telloli added a comment -

          This patch contains code for adding logging on Bookkeeper. This patch by itself requires HDFS-311 and its dependencies to be applied first and it's mainly for review and test.
          To simplify things, people interested in the Bookkeeper functionality can simply download the patch marked with -complete in the HDFS-311 which contains all the dependencies and applies against trunk.

          Show
          Luca Telloli added a comment - This patch contains code for adding logging on Bookkeeper. This patch by itself requires HDFS-311 and its dependencies to be applied first and it's mainly for review and test. To simplify things, people interested in the Bookkeeper functionality can simply download the patch marked with -complete in the HDFS-311 which contains all the dependencies and applies against trunk.
          Hide
          Luca Telloli added a comment -

          A patch for this JIRA has been provided inside HDFS-311. People interested in this patch should take a look to that one. If needed I'll split the two parts.

          Show
          Luca Telloli added a comment - A patch for this JIRA has been provided inside HDFS-311 . People interested in this patch should take a look to that one. If needed I'll split the two parts.
          Hide
          Luca Telloli added a comment -

          Linking to HADOOP-6001, which is required to have all EditLogInput/OutputStream to be then able to process a URI as argument.

          Show
          Luca Telloli added a comment - Linking to HADOOP-6001 , which is required to have all EditLogInput/OutputStream to be then able to process a URI as argument.
          Hide
          Luca Telloli added a comment -

          Setting dependency on HADOOP-5832

          Show
          Luca Telloli added a comment - Setting dependency on HADOOP-5832
          Hide
          Luca Telloli added a comment -

          renamed the .19 version since now we're only working on trunk

          Show
          Luca Telloli added a comment - renamed the .19 version since now we're only working on trunk
          Hide
          Luca Telloli added a comment -

          Newest patch supports reading from Bookkeeper as first source, so a configuration like:
          <property>
          <name>hdfs.editlog</name>
          <value>BOOKKEEPER,FILE</value>
          </property>

          would work and try to read from Bookkeeper first.

          Show
          Luca Telloli added a comment - Newest patch supports reading from Bookkeeper as first source, so a configuration like: <property> <name>hdfs.editlog</name> <value>BOOKKEEPER,FILE</value> </property> would work and try to read from Bookkeeper first.
          Hide
          Luca Telloli added a comment -

          Patch for version .19 breaks the file-based logging system. Patches against trunk (the latest ones) DO NOT.

          Show
          Luca Telloli added a comment - Patch for version .19 breaks the file-based logging system. Patches against trunk (the latest ones) DO NOT.
          Hide
          Luca Telloli added a comment -

          Ben, the current patch needs a storage directory to retrieve the edits. In previous patches I kept this but starting from this one I"m not using storage directories anymore. Thus, when Iogging on BK, there won't be any storage directory available when reading from bookkeeper so nothing to read.
          In a future version I'll get rid of the requirement of having a storage directory to restore edits from, so possibly any logging system should be ok as soon as a stream to read from is provided.

          Show
          Luca Telloli added a comment - Ben, the current patch needs a storage directory to retrieve the edits. In previous patches I kept this but starting from this one I"m not using storage directories anymore. Thus, when Iogging on BK, there won't be any storage directory available when reading from bookkeeper so nothing to read. In a future version I'll get rid of the requirement of having a storage directory to restore edits from, so possibly any logging system should be ok as soon as a stream to read from is provided.
          Hide
          Flavio Junqueira added a comment -

          I've fixed the compilation problem that Konstantin has pointed out. Thanks, Konstantin.

          Show
          Flavio Junqueira added a comment - I've fixed the compilation problem that Konstantin has pointed out. Thanks, Konstantin.
          Hide
          Konstantin Shvachko added a comment -
          1. I've got a compile error:
            [javac] hadoop/src/hdfs/org/apache/hadoop/hdfs/server/namenode/BackupStorage.java:29: cannot find symbol
            [javac] symbol  : class EditLogFileInputStream
            [javac] location: class org.apache.hadoop.hdfs.server.namenode.FSEditLog
            [javac] import org.apache.hadoop.hdfs.server.namenode.FSEditLog.EditLogFileInputStream;
            
          2. It is better to create a separate jira for factoring out EditLogFileOutputStream and EditLogFileInputStream from FSEditLog. This makes sense whether BookKeeper or not, and it will help not to obscure changes you really do to the code.
          3. Why do you need a new method setStorageDirectories() with a Boolean parameter, which is not used anywhere inside.
          4. We will need some automation, which will add zookeeper and bookkeeper jars to the project and synchronize them with new releases.
            Can it be done with ivy?
          5. I agree that edits input part of the code is not generalized for input streams other than EditLogFileInputStream. This is because there were no alternatives yet. We should work on it.

          The drawback of the approach you implement, besides that it requires separate image and edits directories, which you mention, is that you do not have a way to retrieve the latest checkpoint time from the BookKeeper. This is critical for choosing the latest version of the journal, and you can only get the latest checkpoint time from the local file (StorageDirectory) that corresponds to the stream. The StorageDirectory may be out of sync with the real state of the BookKeeper data.

          Suppose that you use one file output stream and one BKOutputStream.
          Suppose the bookKeeper output stream dies, the name-node keeps writing to the file output stream for another hour or so, and then gets restarted.
          If name-node configured to read from the bookKeeper input stream, then it will get an outdated state of the namespace, because the current state is in the local file not in the BK.

          In general I am very glad that this is moving in the right direction and we will eventually have a framework which will allow to plug in different logging systems and intermix them if necessary.

          Show
          Konstantin Shvachko added a comment - I've got a compile error: [javac] hadoop/src/hdfs/org/apache/hadoop/hdfs/server/namenode/BackupStorage.java:29: cannot find symbol [javac] symbol : class EditLogFileInputStream [javac] location: class org.apache.hadoop.hdfs.server.namenode.FSEditLog [javac] import org.apache.hadoop.hdfs.server.namenode.FSEditLog.EditLogFileInputStream; It is better to create a separate jira for factoring out EditLogFileOutputStream and EditLogFileInputStream from FSEditLog . This makes sense whether BookKeeper or not, and it will help not to obscure changes you really do to the code. Why do you need a new method setStorageDirectories() with a Boolean parameter, which is not used anywhere inside. We will need some automation, which will add zookeeper and bookkeeper jars to the project and synchronize them with new releases. Can it be done with ivy? I agree that edits input part of the code is not generalized for input streams other than EditLogFileInputStream. This is because there were no alternatives yet. We should work on it. The drawback of the approach you implement, besides that it requires separate image and edits directories, which you mention, is that you do not have a way to retrieve the latest checkpoint time from the BookKeeper. This is critical for choosing the latest version of the journal, and you can only get the latest checkpoint time from the local file (StorageDirectory) that corresponds to the stream. The StorageDirectory may be out of sync with the real state of the BookKeeper data. Suppose that you use one file output stream and one BKOutputStream. Suppose the bookKeeper output stream dies, the name-node keeps writing to the file output stream for another hour or so, and then gets restarted. If name-node configured to read from the bookKeeper input stream, then it will get an outdated state of the namespace, because the current state is in the local file not in the BK. In general I am very glad that this is moving in the right direction and we will eventually have a framework which will allow to plug in different logging systems and intermix them if necessary.
          Hide
          Benjamin Reed added a comment -

          looks good luca. if curious about your comment about reads being file based. if you configure bookkeeper to be the only logging device, wouldn't it read from bookkeeper?

          i'm also wondering about having to create a new Configuration object to get config parameters. shouldn't a configuration object be passed to the EditLogStreams?

          Show
          Benjamin Reed added a comment - looks good luca. if curious about your comment about reads being file based. if you configure bookkeeper to be the only logging device, wouldn't it read from bookkeeper? i'm also wondering about having to create a new Configuration object to get config parameters. shouldn't a configuration object be passed to the EditLogStreams?
          Hide
          Luca Telloli added a comment - - edited

          I'm posting a new preview version that addresses two features:

          • Logging on multiple devices
          • Writing IDs on Zookeeper (that is, no longer usage of files to write information)

          I additionally moved EditLogFileOutputStream and EditLogFileInputStream out of the FSEditLog class.

          A sample configuration is the following:
          <property>
          <name>dfs.name.dir</name>
          <value>/tmp/localhdfs</value>
          </property>
          <property>
          <name>dfs.name.edits.dir</name>
          <value>/tmp/hdfsedits</value>
          </property>
          <property>
          <name>hdfs.editlog</name>
          <value>FILE,BOOKKEEPER</value>
          </property>

          NOTE: The hdfs.editlog is a new property that has to be specified for this patch to work.

          RUNNING ZOOKEEPER AND BOOKKEEPER EASILY
          To run ZooKeeper and BookKeeper in one shot, there' a class in the bookkeeper .jar named org.apache.bookkeeper.util.LocalBookKeeper which can run a ZooKeeper along with a user-specified number of BookKeepers.

          An example command is the following:
          java -cp lib/log4j-1.2.15.jar:lib/junit-3.8.1.jar:lib/zookeeper-dev.jar:lib/zookeeper-dev-bookkeeper.jar org.apache.bookkeeper.util.LocalBookKeeper N

          where N is the number of Bookies to use (for instance 3)

          LOGGING ON MULTIPLE DEVICES
          The initial semantic is very simple, and is the following:

          • when writing an operation, write sequentially to all types of logging
          • when reading operations (during the startup or checkpoint), read from the first logging system; at the moment this is the first storage directory, so still file-based

          There's no fall-back mechanism implemented yet if the first logging system fails (the idea would be to go with the next one and exclude the failed one from the array of streams).

          The current loadFSEdits(StorageDirectory) should eventually change to a loadFSEdits() where no storage directory is needed. Maybe a loadFSEdits(EditLogInputStream) would be even better.

          DRAWBACKS
          Currently, storage directories can be of three types: IMAGE, EDITS and IMAGE_AND_EDITS, with the last one being the default one. With this patch I exclude the IMAGE_AND_EDITS type, so user are forced to use the dfs.name.dir and dfs.name.edits.dir to specify a directory for IMAGE and a directory for EDITS, when using file logging.

          Show
          Luca Telloli added a comment - - edited I'm posting a new preview version that addresses two features: Logging on multiple devices Writing IDs on Zookeeper (that is, no longer usage of files to write information) I additionally moved EditLogFileOutputStream and EditLogFileInputStream out of the FSEditLog class. A sample configuration is the following: <property> <name>dfs.name.dir</name> <value>/tmp/localhdfs</value> </property> <property> <name>dfs.name.edits.dir</name> <value>/tmp/hdfsedits</value> </property> <property> <name>hdfs.editlog</name> <value>FILE,BOOKKEEPER</value> </property> NOTE: The hdfs.editlog is a new property that has to be specified for this patch to work. RUNNING ZOOKEEPER AND BOOKKEEPER EASILY To run ZooKeeper and BookKeeper in one shot, there' a class in the bookkeeper .jar named org.apache.bookkeeper.util.LocalBookKeeper which can run a ZooKeeper along with a user-specified number of BookKeepers. An example command is the following: java -cp lib/log4j-1.2.15.jar:lib/junit-3.8.1.jar:lib/zookeeper-dev.jar:lib/zookeeper-dev-bookkeeper.jar org.apache.bookkeeper.util.LocalBookKeeper N where N is the number of Bookies to use (for instance 3) LOGGING ON MULTIPLE DEVICES The initial semantic is very simple, and is the following: when writing an operation, write sequentially to all types of logging when reading operations (during the startup or checkpoint), read from the first logging system; at the moment this is the first storage directory, so still file-based There's no fall-back mechanism implemented yet if the first logging system fails (the idea would be to go with the next one and exclude the failed one from the array of streams). The current loadFSEdits(StorageDirectory) should eventually change to a loadFSEdits() where no storage directory is needed. Maybe a loadFSEdits(EditLogInputStream) would be even better. DRAWBACKS Currently, storage directories can be of three types: IMAGE, EDITS and IMAGE_AND_EDITS, with the last one being the default one. With this patch I exclude the IMAGE_AND_EDITS type, so user are forced to use the dfs.name.dir and dfs.name.edits.dir to specify a directory for IMAGE and a directory for EDITS, when using file logging.
          Hide
          Luca Telloli added a comment -
          • This patch applies against trunk
          • In this patch the secondary namenode is not patched since the architecture changed; thus at the moment only the namenode works
          • the patch does not use zookeeper yet to store information. it's on the works
          • the patch does not currently support multiple concurrent logging systems, it's on the works as well
          Show
          Luca Telloli added a comment - This patch applies against trunk In this patch the secondary namenode is not patched since the architecture changed; thus at the moment only the namenode works the patch does not use zookeeper yet to store information. it's on the works the patch does not currently support multiple concurrent logging systems, it's on the works as well
          Hide
          Benjamin Reed added a comment -

          great job luca! konstantin and dhruba can we get some direction on this. there needs to be a more abstract implementation agnostic interface to the edit log and image.

          Show
          Benjamin Reed added a comment - great job luca! konstantin and dhruba can we get some direction on this. there needs to be a more abstract implementation agnostic interface to the edit log and image.
          Hide
          Luca Telloli added a comment - - edited

          I'm posting a new patch for the integration of BookKeeper with HDFS.

          In this patch the logSync() method is exactly the same as the original file based logging. Additionally, it does not implement any abstract class for logging, apart from the two EditLogInput/OutputStream classes, as requested by Konstantin (in the following I'll just use InputStream, but I'll refer to both).

          Here's some detail about the patch:

          1. the current implementation, HDFS does not allow other types of logging; specifically version .19 does not allow any other EditLogInputStream apart from EditLogFileInputStream, that is, each time a EditLogInputStream is needed, a EditLogFileInputStream is instantiated. In the patch I add configuration values to enable Bookkeper logging and I allow the user to switch between different logging types by using a configuration property in hadoop-site.xml

          2. As Konstantin suggested some time ago, in the current patch I started by implementing only the two abstract classes above, but at the end I had to modify more classes. In particular I modify FSEditLog.java, SecondaryNamenode.java and FSImage.java. Although the modifications are mainly related to issue 1, there's an additional confusion between the semantics of "open" and "create" since, in the case of files, the two operations have strong similarities. This doesn't hold for BookKeeper in some cases (mostly related to the CreateEditLogFile() method), my code needs to branch to avoid some unwanted creation of new ledgers.

          Even with this, the patch is not yet complete, due to the following issues:

          3. the current .19 implementation does not yet implement support for multiple concurrent logging systems. This is another implementation problem which should be fixed, but I'm not sure how easily. As Ben said, HDFS is heavily based on file and uses storage directories to store the image of the file system and the edits in these directories. I think this then turns into a design problem, because it's not easy to decouple the file system from the edits file, since they both live in the same directory.

          4. Another drawback related to 3 is that the Namenode, to properly work, needs some files like edits and edits new even if the logging system is not using file-based logging. In the patch, even if I'm using Bookkeeper to store edits, I still need to have the edits and edits.new files. I currently use them to store some small information about ledgers IDs but this will change soon in favor of ZooKeeper.

          I'm not sure how to fix the above issues, in particular I'm worried that a good solution would need to rethink the Storage Directories and the FSImage as they're currently implemented.

          Show
          Luca Telloli added a comment - - edited I'm posting a new patch for the integration of BookKeeper with HDFS. In this patch the logSync() method is exactly the same as the original file based logging. Additionally, it does not implement any abstract class for logging, apart from the two EditLogInput/OutputStream classes, as requested by Konstantin (in the following I'll just use InputStream, but I'll refer to both). Here's some detail about the patch: 1. the current implementation, HDFS does not allow other types of logging; specifically version .19 does not allow any other EditLogInputStream apart from EditLogFileInputStream, that is, each time a EditLogInputStream is needed, a EditLogFileInputStream is instantiated. In the patch I add configuration values to enable Bookkeper logging and I allow the user to switch between different logging types by using a configuration property in hadoop-site.xml 2. As Konstantin suggested some time ago, in the current patch I started by implementing only the two abstract classes above, but at the end I had to modify more classes. In particular I modify FSEditLog.java, SecondaryNamenode.java and FSImage.java. Although the modifications are mainly related to issue 1, there's an additional confusion between the semantics of "open" and "create" since, in the case of files, the two operations have strong similarities. This doesn't hold for BookKeeper in some cases (mostly related to the CreateEditLogFile() method), my code needs to branch to avoid some unwanted creation of new ledgers. Even with this, the patch is not yet complete, due to the following issues: 3. the current .19 implementation does not yet implement support for multiple concurrent logging systems. This is another implementation problem which should be fixed, but I'm not sure how easily. As Ben said, HDFS is heavily based on file and uses storage directories to store the image of the file system and the edits in these directories. I think this then turns into a design problem, because it's not easy to decouple the file system from the edits file, since they both live in the same directory. 4. Another drawback related to 3 is that the Namenode, to properly work, needs some files like edits and edits new even if the logging system is not using file-based logging. In the patch, even if I'm using Bookkeeper to store edits, I still need to have the edits and edits.new files. I currently use them to store some small information about ledgers IDs but this will change soon in favor of ZooKeeper. I'm not sure how to fix the above issues, in particular I'm worried that a good solution would need to rethink the Storage Directories and the FSImage as they're currently implemented.
          Hide
          Flavio Junqueira added a comment -

          Dhruba, The BookKeeper API does not restrict the application to read in a given order. The API allows an application to request a sequence of entries x1..xn, so if the application requests such sequences in order, it will receive entries in order.

          Show
          Flavio Junqueira added a comment - Dhruba, The BookKeeper API does not restrict the application to read in a given order. The API allows an application to request a sequence of entries x1..xn, so if the application requests such sequences in order, it will receive entries in order.
          Hide
          dhruba borthakur added a comment -

          > if the namenode adds entry x and later y, the bookkeeper client won't acknowledge y before x to the namenode.

          Also, does this mean that when the ledger is being read later on, x is guaranteed to appear before y?

          Show
          dhruba borthakur added a comment - > if the namenode adds entry x and later y, the bookkeeper client won't acknowledge y before x to the namenode. Also, does this mean that when the ledger is being read later on, x is guaranteed to appear before y?
          Hide
          Luca Telloli added a comment -

          Raghu, I'll reply on HADOOP-5188 since it's more about that, ok? Thanks

          Show
          Luca Telloli added a comment - Raghu, I'll reply on HADOOP-5188 since it's more about that, ok? Thanks
          Hide
          Raghu Angadi added a comment -

          btw, I didn't mean to change the direction of the effort or the patch. I think it is ok to include HADOOP-5188 for now.

          Show
          Raghu Angadi added a comment - btw, I didn't mean to change the direction of the effort or the patch. I think it is ok to include HADOOP-5188 for now.
          Hide
          Raghu Angadi added a comment -

          Thanks.

          Regd ordering, I guess acks are internal to BookKeeper adapter. All NameNode needs is that the records are read/returned in the order it wrote.

          About HADOOP-5188, it not very clear if the current interface is terribly inadequate (that does not mean it cann't be improved). Roughly, the BookKeeper adapter needs to extend 'EditLogInputStream' and 'EditLogOutputStream'. For. e.g. 'setReadyToFlush()" might be no-op and 'flushAndSync()' just waits for ack for the latest record (at the time of the call)... Note that NN is writing more records in parallel even while it is waiting for flushAndSync()'.

          Show
          Raghu Angadi added a comment - Thanks. Regd ordering, I guess acks are internal to BookKeeper adapter. All NameNode needs is that the records are read/returned in the order it wrote. About HADOOP-5188 , it not very clear if the current interface is terribly inadequate (that does not mean it cann't be improved). Roughly, the BookKeeper adapter needs to extend 'EditLogInputStream' and 'EditLogOutputStream'. For. e.g. 'setReadyToFlush()" might be no-op and 'flushAndSync()' just waits for ack for the latest record (at the time of the call)... Note that NN is writing more records in parallel even while it is waiting for flushAndSync()'.
          Hide
          Flavio Junqueira added a comment -

          Hi Raghu, The patch for this issue currently includes HADOOP-5188. We couldn't find a clean way of separating the patches, but we felt it is worth discussing the issues separately. Please feel free to suggest a different approach if there is a better way of doing it.

          With respect to ordering, BookKeeper does respect the order of add calls, even if the calls are asynchronous. That is, if the namenode adds entry x and later y, the bookkeeper client won't acknowledge y before x to the namenode.

          Show
          Flavio Junqueira added a comment - Hi Raghu, The patch for this issue currently includes HADOOP-5188 . We couldn't find a clean way of separating the patches, but we felt it is worth discussing the issues separately. Please feel free to suggest a different approach if there is a better way of doing it. With respect to ordering, BookKeeper does respect the order of add calls, even if the calls are asynchronous. That is, if the namenode adds entry x and later y, the bookkeeper client won't acknowledge y before x to the namenode.
          Hide
          dhruba borthakur added a comment -

          > I guess this could still be part of HADOOP-5188

          Yes, i think you are right. let's continue that discussion as part of HADOOP-5188. Will it be fair to mark HADOOP-5188 as a pre-requisite for HADOOP-5189?

          Show
          dhruba borthakur added a comment - > I guess this could still be part of HADOOP-5188 Yes, i think you are right. let's continue that discussion as part of HADOOP-5188 . Will it be fair to mark HADOOP-5188 as a pre-requisite for HADOOP-5189 ?
          Hide
          Luca Telloli added a comment -

          hey Dhruba, following your suggestion I thought about having multiple logging systems in parallel and the simplest solution I could think of is a vector of EditLogs inside FSImage. How does it sound?

          As I imagine it, this would require a small amount of code changes, would not add too much overhead, would allow the user to specify multiple types of logging at the same time if needed and easily fall back to a case with a single logger. I don't see many issues for having it working without issues, apart from some performance penalty due to the slower synchronization process. I guess this could still be part of HADOOP-5188?

          Show
          Luca Telloli added a comment - hey Dhruba, following your suggestion I thought about having multiple logging systems in parallel and the simplest solution I could think of is a vector of EditLogs inside FSImage. How does it sound? As I imagine it, this would require a small amount of code changes, would not add too much overhead, would allow the user to specify multiple types of logging at the same time if needed and easily fall back to a case with a single logger. I don't see many issues for having it working without issues, apart from some performance penalty due to the slower synchronization process. I guess this could still be part of HADOOP-5188 ?
          Hide
          Raghu Angadi added a comment -

          Does this patch include HADOOP-5188 as well? There seem to be a lot of 'while space only' changes in the patch.

          If the records are written asynchronously to book keeper, is the ordering controlled explicitly (by the NameNode side Bookkeeper client)?

          I agree interface should allow edit log implementation to do decide how to wait for sync, but ordering is probably a NameNode level requirement.

          Show
          Raghu Angadi added a comment - Does this patch include HADOOP-5188 as well? There seem to be a lot of 'while space only' changes in the patch. If the records are written asynchronously to book keeper, is the ordering controlled explicitly (by the NameNode side Bookkeeper client)? I agree interface should allow edit log implementation to do decide how to wait for sync, but ordering is probably a NameNode level requirement.
          Hide
          dhruba borthakur added a comment -

          Hi Luca, Thanks for your comments. It would be nice if we can enhance the current Namenode code to support invoking logSync() from multiple threads. This will help namenode performance even when it is writing edits-log to multiple files (and no bookKeeper is involed). If we have a patch for HADOOP-5188 and we apply that first to the Hadoop trunk, then the bookKeeper patch will fit very elegantly.

          If the above is done as a pre-requisite for your bookKeeper patch, then we would be able to support file based edits log as well as bookKeeper based transaction log as the same time. This means that a namenode can log each transactions to a local file as well as bookKeeper. This might help in transitioning current hdfs namenode to using bookKeeper.

          > it's more complex and not sure if it's worth the effort, in principle because you'd have to maintain two different logging system in parallel

          I agree.

          Show
          dhruba borthakur added a comment - Hi Luca, Thanks for your comments. It would be nice if we can enhance the current Namenode code to support invoking logSync() from multiple threads. This will help namenode performance even when it is writing edits-log to multiple files (and no bookKeeper is involed). If we have a patch for HADOOP-5188 and we apply that first to the Hadoop trunk, then the bookKeeper patch will fit very elegantly. If the above is done as a pre-requisite for your bookKeeper patch, then we would be able to support file based edits log as well as bookKeeper based transaction log as the same time. This means that a namenode can log each transactions to a local file as well as bookKeeper. This might help in transitioning current hdfs namenode to using bookKeeper. > it's more complex and not sure if it's worth the effort, in principle because you'd have to maintain two different logging system in parallel I agree.
          Hide
          Luca Telloli added a comment -

          Dhruba, I found the EditLogInput/OutputStream interfaces insufficient for what we wanted to achieve with BookKeeper, and because of this I separately filed https://issues.apache.org/jira/browse/HADOOP-5188 .
          In this BookKeeper patch I modify the LogSync() method to allow parallel asynchronous writing from multiple threads, while in the current logging system there's a single thread writing at any time. We couldn't achieve as good results as we have with just a single thread, thus the need for 5188.

          With respect to maintaining the file-based logging I'm not sure if you mean the ability to switch from one way of logging to the other, or for having them in parallel. The second one I think it's more complex and not sure if it's worth the effort, in principle because you'd have to maintain two different logging system in parallel and ensure that they are both consistent. If you mean the first one, I'm thinking about a fix.

          Show
          Luca Telloli added a comment - Dhruba, I found the EditLogInput/OutputStream interfaces insufficient for what we wanted to achieve with BookKeeper, and because of this I separately filed https://issues.apache.org/jira/browse/HADOOP-5188 . In this BookKeeper patch I modify the LogSync() method to allow parallel asynchronous writing from multiple threads, while in the current logging system there's a single thread writing at any time. We couldn't achieve as good results as we have with just a single thread, thus the need for 5188. With respect to maintaining the file-based logging I'm not sure if you mean the ability to switch from one way of logging to the other, or for having them in parallel. The second one I think it's more complex and not sure if it's worth the effort, in principle because you'd have to maintain two different logging system in parallel and ensure that they are both consistent. If you mean the first one, I'm thinking about a fix.
          Hide
          dhruba borthakur added a comment -

          This feature is cool!

          From a migration-of-existing-clusters point of view, it would be nice if the namenode can continue to log its edits log to a local file as well as write to book-keeper. This ensures that existing clusters can plug in a book-keeper instance, build confidence in the system over time, and then completely migrate to bookeeper. In the current hadoop trunk, there is an Interface called EditLogOutputStream that is used to integrate other storage software to log edit streams. If this patch implements the Interface org.apache.hadoop.hdfs.server.namenode.EditLogOutputStream, then the namenode can log its edits log into a file as well as bookKeeper.

          Show
          dhruba borthakur added a comment - This feature is cool! From a migration-of-existing-clusters point of view, it would be nice if the namenode can continue to log its edits log to a local file as well as write to book-keeper. This ensures that existing clusters can plug in a book-keeper instance, build confidence in the system over time, and then completely migrate to bookeeper. In the current hadoop trunk, there is an Interface called EditLogOutputStream that is used to integrate other storage software to log edit streams. If this patch implements the Interface org.apache.hadoop.hdfs.server.namenode.EditLogOutputStream, then the namenode can log its edits log into a file as well as bookKeeper.
          Hide
          Luca Telloli added a comment -

          Jeff, I think that for being a logging abstraction, BookKeeper does not implement it but provides a way to achieve it. That is, if the NameNode dies, another process could reconstruct the fsimage starting from a set of ledgers. Actually, any number of independent processes could do it, as soon as they read from the same set of ledgers. This process could definitely be the SecondaryNameNode.
          I guess that such a process could be implemented to allow warm standby, as proposed in the description. First thought: ZooKeeper could be used to store such an information (the set of ledger IDs that belong to a single fsimage) and the secondary NameNodes could periodically poll for changes (or maybe using some notification mechanism).

          Show
          Luca Telloli added a comment - Jeff, I think that for being a logging abstraction, BookKeeper does not implement it but provides a way to achieve it. That is, if the NameNode dies, another process could reconstruct the fsimage starting from a set of ledgers. Actually, any number of independent processes could do it, as soon as they read from the same set of ledgers. This process could definitely be the SecondaryNameNode. I guess that such a process could be implemented to allow warm standby, as proposed in the description. First thought: ZooKeeper could be used to store such an information (the set of ledger IDs that belong to a single fsimage) and the secondary NameNodes could periodically poll for changes (or maybe using some notification mechanism).
          Hide
          Jeff Hammerbacher added a comment -

          Hey Luca,

          Could you comment on how this patch relates to HADOOP-4539?

          Thanks,
          Jeff

          Show
          Jeff Hammerbacher added a comment - Hey Luca, Could you comment on how this patch relates to HADOOP-4539 ? Thanks, Jeff
          Hide
          Luca Telloli added a comment -

          Attaching first patch

          Show
          Luca Telloli added a comment - Attaching first patch
          Hide
          Luca Telloli added a comment -

          submitted by mistake

          Show
          Luca Telloli added a comment - submitted by mistake
          Hide
          Luca Telloli added a comment -

          Attaching preliminary results graph

          Show
          Luca Telloli added a comment - Attaching preliminary results graph
          Hide
          Luca Telloli added a comment -

          Integrating Bookkeeper into HDFS
          --------------------------------

          INTRO
          ------
          BookKeeper is a system to reliably log streams of records (https://issues.apache.org/jira/browse/ZOOKEEPER-276). The NameNode is a natural target for such a system for being the metadata repository of the entire file system for HDFS.

          The NameNode works by logging any modification to the file system on a separate stream and periodically merge the stream with the latest image of the file system. The standard version of HDFS only supports file-based logging on any number of devices. This patch provides BookKeeper logging.

          The advantages of BookKeeper-logging over file-logging are:

          • higher throughput performance in large deployments
          • higher availability though externally recoverable log files (ledgers)

          This patch provides:

          HOW TO APPLY THE PATCH
          -------------------------
          The patch is available against Hadoop release 0.19 (http://svn.apache.org/repos/asf/hadoop/core/tags/release-0.19.0/). This patch includes https://issues.apache.org/jira/browse/HADOOP-5188 (Modifications to enable multiple types of logging)

          The patch does not support file-based logging, nor it offers a way to switch from one system to the other, so please apply with caution. The patch does not support HDFS deployment older than 0.19 and has been tested only over pre-formatted file-systems; although it should work without many troubles, we suggest not to apply this patch to production systems.

          To compile, just run ant. To compile and run properly you will need to add the zookeeper and bookkeeper jar to you hadoop/lib directory. Please refer to http://hadoop.apache.org/zookeeper/ for information on how to get, compile and run Zookeeper and Bookkeeper.

          HOW TO USE THE PATCHED HDFS
          ----------------------------
          One you patch Hadoop there will be some new properties to configure in hadoop-site.xml; a description follows:

          <property>
          <!--
          Specifies the type of logging to use. At the moment, only BKEditLogThreadBuf is provided.
          -->
          <name>hdfs.editlog</name>
          <value>BKEditLogThreadBuf</value>
          </property>

          <property>
          <!--
          Specifies the number of bookies to use. Default is 3.
          -->
          <name>bklog.bookies.total</name>
          <value>3</value>
          </property>

          <property>
          <!--
          Specifies the size of the quorum. Default is 2.
          -->
          <name>bklog.bookies.quorumsize</name>
          <value>2</value>
          </property>

          <property>
          <!--
          Specifies the logging mode. A string in

          {"verifiable", "generic"}

          . For more information on this see TODO
          -->
          <name>bklog.bookies.qmode</name>
          <value>verifiable</value>
          </property>

          <property>
          <!--
          Specifies the ZooKeeper server containing info about bookies
          -->
          <name>bklog.zookeeper.servers</name>
          <value>127.0.0.1</value>
          </property>

          Once HDFS is configured, the subsequent steps are:

          PERFORMANCE
          -----------
          We tested our prototype against vanilla HDFS using a benchmark provided by the Hadoop folks and
          available in the Hadoop distribution as org.apache.hadoop.hdfs.NNThroughputBenchmark. BookKeeper
          logging was configured to use 3 bookies with a quorum of 2 in the verifiable mode. File-logging was
          configured to write on a single device (FS) or two devices (FS-NFS, one of them being mounted through NFS).

          The machines were dual core 4G with 2 300G SATA drives.

          The variables involved are the number of concurrent threads and the number of operations to log. We tested logging 400k ops of type create, and the throughput (measured in Ops/s) is depicted in the attached graph.

          Show
          Luca Telloli added a comment - Integrating Bookkeeper into HDFS -------------------------------- INTRO ------ BookKeeper is a system to reliably log streams of records ( https://issues.apache.org/jira/browse/ZOOKEEPER-276 ). The NameNode is a natural target for such a system for being the metadata repository of the entire file system for HDFS. The NameNode works by logging any modification to the file system on a separate stream and periodically merge the stream with the latest image of the file system. The standard version of HDFS only supports file-based logging on any number of devices. This patch provides BookKeeper logging. The advantages of BookKeeper-logging over file-logging are: higher throughput performance in large deployments higher availability though externally recoverable log files (ledgers) This patch provides: a new abstraction for logging (also filed as https://issues.apache.org/jira/browse/HADOOP-5188 ) an integration between Hadoop's HDFS and BookKeeper HOW TO APPLY THE PATCH ------------------------- The patch is available against Hadoop release 0.19 ( http://svn.apache.org/repos/asf/hadoop/core/tags/release-0.19.0/ ). This patch includes https://issues.apache.org/jira/browse/HADOOP-5188 (Modifications to enable multiple types of logging) The patch does not support file-based logging, nor it offers a way to switch from one system to the other, so please apply with caution. The patch does not support HDFS deployment older than 0.19 and has been tested only over pre-formatted file-systems; although it should work without many troubles, we suggest not to apply this patch to production systems. To compile, just run ant. To compile and run properly you will need to add the zookeeper and bookkeeper jar to you hadoop/lib directory. Please refer to http://hadoop.apache.org/zookeeper/ for information on how to get, compile and run Zookeeper and Bookkeeper. HOW TO USE THE PATCHED HDFS ---------------------------- One you patch Hadoop there will be some new properties to configure in hadoop-site.xml; a description follows: <property> <!-- Specifies the type of logging to use. At the moment, only BKEditLogThreadBuf is provided. --> <name>hdfs.editlog</name> <value>BKEditLogThreadBuf</value> </property> <property> <!-- Specifies the number of bookies to use. Default is 3. --> <name>bklog.bookies.total</name> <value>3</value> </property> <property> <!-- Specifies the size of the quorum. Default is 2. --> <name>bklog.bookies.quorumsize</name> <value>2</value> </property> <property> <!-- Specifies the logging mode. A string in {"verifiable", "generic"} . For more information on this see TODO --> <name>bklog.bookies.qmode</name> <value>verifiable</value> </property> <property> <!-- Specifies the ZooKeeper server containing info about bookies --> <name>bklog.zookeeper.servers</name> <value>127.0.0.1</value> </property> Once HDFS is configured, the subsequent steps are: start ZooKeeper start the bookies initialise ZooKeeper with the bookies' host ( ZOOKEEPER-301 https://issues.apache.org/jira/browse/ZOOKEEPER-301 provides a util class for this) start HDFS ready to GO! PERFORMANCE ----------- We tested our prototype against vanilla HDFS using a benchmark provided by the Hadoop folks and available in the Hadoop distribution as org.apache.hadoop.hdfs.NNThroughputBenchmark. BookKeeper logging was configured to use 3 bookies with a quorum of 2 in the verifiable mode. File-logging was configured to write on a single device (FS) or two devices (FS-NFS, one of them being mounted through NFS). The machines were dual core 4G with 2 300G SATA drives. The variables involved are the number of concurrent threads and the number of operations to log. We tested logging 400k ops of type create, and the throughput (measured in Ops/s) is depicted in the attached graph.

            People

            • Assignee:
              Ivan Kelly
              Reporter:
              Luca Telloli
            • Votes:
              0 Vote for this issue
              Watchers:
              29 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development