Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-2743

Streamline usage of bookkeeper journal manager

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.24.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Target Version/s:

      Description

      The current method of installing bkjournal manager involves generating a tarball, and extracting it with special flags over the hdfs distribution. This is cumbersome and prone to being broken by other changes (see https://svn.apache.org/repos/asf/hadoop/common/trunk@1220940). I think a cleaner way to doing this is to generate a single jar that can be placed in the lib dir of hdfs.

      1. HDFS-2743.diff
        4 kB
        Ivan Kelly
      2. HDFS-2743.diff
        3 kB
        Ivan Kelly

        Activity

        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12509301/HDFS-2743.diff
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        -1 javadoc. The javadoc tool appears to have generated 20 warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

        -1 release audit. The applied patch generated 1 release audit warnings (more than the trunk's current 0 warnings).

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.hdfs.TestFileAppend2

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1749//testReport/
        Release audit warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1749//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1749//artifact/trunk/hadoop-hdfs-project/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1749//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12509301/HDFS-2743.diff against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 javadoc. The javadoc tool appears to have generated 20 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. -1 release audit. The applied patch generated 1 release audit warnings (more than the trunk's current 0 warnings). -1 core tests. The patch failed these unit tests: org.apache.hadoop.hdfs.TestFileAppend2 +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1749//testReport/ Release audit warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1749//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1749//artifact/trunk/hadoop-hdfs-project/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1749//console This message is automatically generated.
        Hide
        Ivan Kelly added a comment -
        {qoute}

        -1 tests included. The patch doesn't appear to include any new or modified tests.

        Change is to packaging, no test required.

        -1 javadoc. The javadoc tool appears to have generated 20 warning messages.
        -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

        This are not within the bkjournal contrib module.

        -1 release audit. The applied patch generated 1 release audit warnings (more than the trunk's current 0 warnings).

        This relates to: <br/>
        !????? /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/dev-support/findbugsExcludeFile.xml

        Show
        Ivan Kelly added a comment - {qoute} -1 tests included. The patch doesn't appear to include any new or modified tests. Change is to packaging, no test required. -1 javadoc. The javadoc tool appears to have generated 20 warning messages. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. This are not within the bkjournal contrib module. -1 release audit. The applied patch generated 1 release audit warnings (more than the trunk's current 0 warnings). This relates to: <br/> !????? /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/dev-support/findbugsExcludeFile.xml
        Hide
        Todd Lipcon added a comment -

        This doesn't look quite right. The jar-with-dependencies has too many dependencies, including junit and extra copies of some things that Hadoop already depends on (eg commons-math, commons-logging, log4j, etc). It should include only the minimal dependencies, and probably better off shading them as well IMO.

        Show
        Todd Lipcon added a comment - This doesn't look quite right. The jar-with-dependencies has too many dependencies, including junit and extra copies of some things that Hadoop already depends on (eg commons-math, commons-logging, log4j, etc). It should include only the minimal dependencies, and probably better off shading them as well IMO.
        Hide
        Ivan Kelly added a comment -

        Ah, you are right. I've updated the patch to only pull in zookeeper, bookkeeper and netty. All 3 are relocated also, so as not to interfere with anything else.

        Show
        Ivan Kelly added a comment - Ah, you are right. I've updated the patch to only pull in zookeeper, bookkeeper and netty. All 3 are relocated also, so as not to interfere with anything else.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12509923/HDFS-2743.diff
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        -1 javadoc. The javadoc tool appears to have generated 21 warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

        -1 release audit. The applied patch generated 1 release audit warnings (more than the trunk's current 0 warnings).

        +1 core tests. The patch passed unit tests in .

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1768//testReport/
        Release audit warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1768//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1768//artifact/trunk/hadoop-hdfs-project/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1768//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12509923/HDFS-2743.diff against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 javadoc. The javadoc tool appears to have generated 21 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. -1 release audit. The applied patch generated 1 release audit warnings (more than the trunk's current 0 warnings). +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1768//testReport/ Release audit warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1768//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1768//artifact/trunk/hadoop-hdfs-project/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1768//console This message is automatically generated.
        Hide
        Ivan Kelly added a comment -

        As before, warnings are unrelated.

        Show
        Ivan Kelly added a comment - As before, warnings are unrelated.
        Hide
        Uma Maheswara Rao G added a comment -

        Ivan, Patch looks fine for me.
        I have applied this patch and used the genrated BK jar in HA setsup.

        As we know, the other problem still exists.

        2012-04-18 10:39:09,423 ERROR org.apache.bookkeeper.client.PendingReadOp: Error: No such entry while reading entry: 1 ledgerId: 5 from bookie: /10.18.52.157:3181
        2012-04-18 10:39:09,457 WARN org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer: Edits tailer failed to find any streams. Will try again later.
        java.io.IOException: No ledger for fromTxnId -12344 found.
        	at org.apache.hadoop.contrib.bkjournal.BookKeeperJournalManager.getInputStream(BookKeeperJournalManager.java:329)
        	at org.apache.hadoop.hdfs.server.namenode.JournalSet.getInputStream(JournalSet.java:246)
        	at org.apache.hadoop.hdfs.server.namenode.FSEditLog.selectInputStreams(FSEditLog.java:1100)
        	at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.doTailEdits(EditLogTailer.java:206)
        	at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.access$700(EditLogTailer.java:59)
        	at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:318)
        	at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$200(EditLogTailer.java:276)
        	at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:293)
        	at org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:504)
        	at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.run(EditLogTailer.java:289)
        

        I think you already handled them as part of other issues, like HDFS-2717, HDFS-3058.

        I am ok with this patch. I will commit the patch tomorrow, if there are no objections.

        Show
        Uma Maheswara Rao G added a comment - Ivan, Patch looks fine for me. I have applied this patch and used the genrated BK jar in HA setsup. As we know, the other problem still exists. 2012-04-18 10:39:09,423 ERROR org.apache.bookkeeper.client.PendingReadOp: Error: No such entry while reading entry: 1 ledgerId: 5 from bookie: /10.18.52.157:3181 2012-04-18 10:39:09,457 WARN org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer: Edits tailer failed to find any streams. Will try again later. java.io.IOException: No ledger for fromTxnId -12344 found. at org.apache.hadoop.contrib.bkjournal.BookKeeperJournalManager.getInputStream(BookKeeperJournalManager.java:329) at org.apache.hadoop.hdfs.server.namenode.JournalSet.getInputStream(JournalSet.java:246) at org.apache.hadoop.hdfs.server.namenode.FSEditLog.selectInputStreams(FSEditLog.java:1100) at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.doTailEdits(EditLogTailer.java:206) at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.access$700(EditLogTailer.java:59) at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:318) at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$200(EditLogTailer.java:276) at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:293) at org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:504) at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.run(EditLogTailer.java:289) I think you already handled them as part of other issues, like HDFS-2717 , HDFS-3058 . I am ok with this patch. I will commit the patch tomorrow, if there are no objections.
        Hide
        Ivan Kelly added a comment -

        I think you already handled them as part of other issues, like HDFS-2717, HDFS-3058.

        Yup, HA won't work without HDFS-3058

        Show
        Ivan Kelly added a comment - I think you already handled them as part of other issues, like HDFS-2717 , HDFS-3058 . Yup, HA won't work without HDFS-3058
        Hide
        Uma Maheswara Rao G added a comment -

        +1

        Show
        Uma Maheswara Rao G added a comment - +1
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #2223 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2223/)
        HDFS-2743. Streamline usage of bookkeeper journal manager. Contributed by Ivan Kelly. (Revision 1331790)

        Result = SUCCESS
        umamahesh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1331790
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/README.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/pom.xml
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2223 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2223/ ) HDFS-2743 . Streamline usage of bookkeeper journal manager. Contributed by Ivan Kelly. (Revision 1331790) Result = SUCCESS umamahesh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1331790 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/README.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/pom.xml
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #2149 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2149/)
        HDFS-2743. Streamline usage of bookkeeper journal manager. Contributed by Ivan Kelly. (Revision 1331790)

        Result = SUCCESS
        umamahesh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1331790
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/README.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/pom.xml
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2149 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2149/ ) HDFS-2743 . Streamline usage of bookkeeper journal manager. Contributed by Ivan Kelly. (Revision 1331790) Result = SUCCESS umamahesh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1331790 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/README.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/pom.xml
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #2166 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2166/)
        HDFS-2743. Streamline usage of bookkeeper journal manager. Contributed by Ivan Kelly. (Revision 1331790)

        Result = ABORTED
        umamahesh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1331790
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/README.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/pom.xml
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2166 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2166/ ) HDFS-2743 . Streamline usage of bookkeeper journal manager. Contributed by Ivan Kelly. (Revision 1331790) Result = ABORTED umamahesh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1331790 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/README.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/pom.xml
        Hide
        Uma Maheswara Rao G added a comment -

        I have just committed to trunk!
        Thanks a lot, Ivan.

        Show
        Uma Maheswara Rao G added a comment - I have just committed to trunk! Thanks a lot, Ivan.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #1029 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1029/)
        HDFS-2743. Streamline usage of bookkeeper journal manager. Contributed by Ivan Kelly. (Revision 1331790)

        Result = FAILURE
        umamahesh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1331790
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/README.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/pom.xml
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1029 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1029/ ) HDFS-2743 . Streamline usage of bookkeeper journal manager. Contributed by Ivan Kelly. (Revision 1331790) Result = FAILURE umamahesh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1331790 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/README.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/pom.xml
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #1064 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1064/)
        HDFS-2743. Streamline usage of bookkeeper journal manager. Contributed by Ivan Kelly. (Revision 1331790)

        Result = FAILURE
        umamahesh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1331790
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/README.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/pom.xml
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1064 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1064/ ) HDFS-2743 . Streamline usage of bookkeeper journal manager. Contributed by Ivan Kelly. (Revision 1331790) Result = FAILURE umamahesh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1331790 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/README.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/pom.xml

          People

          • Assignee:
            Ivan Kelly
            Reporter:
            Ivan Kelly
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development