Hama
  1. Hama
  2. HAMA-437

PiEstimator is not working in Local Mode

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.3.0
    • Fix Version/s: 0.4.0
    • Component/s: None
    • Labels:
      None

      Description

      While playing arround in HAMA-436, I observed that PiEstimator is not working in local mode because of the peer name manipulation.

      Shortest Paths and Pagerank are working because they use the hostname directly without constructing it with a default port.

      Additionally we should add a TestCase for the LocalBSPRunner that launches the PiEstimator.

      1. HAMA-437_v2.patch
        24 kB
        Thomas Jungblut
      2. HAMA-437.patch
        28 kB
        Thomas Jungblut

        Activity

        Hide
        Thomas Jungblut added a comment -

        I have fixed PiEstimator now, it works in both modes.

        Additionally I have added some outputs when starting a cluster in local mode, because I had some mailing list fun the last days.

        If the apps determine that there is "local" in "bsp.master.adress" then the application exits and prints that it doesn't need to be started.
        This will now prevent the questions: "my error log shows, local is no valid hostname". Although I think this is a pretty clear exception >_>.

        Anyways, you can now submit local jobs via the shell ./hama jar <jar.jar> test for example.

        And I added testcases for the local mode and fixed a bit with the name handling in our cluster status.

        There were a hell lot of other problems, for example a local running BSP that could not work on DFS because the filename wasn't qualified with hdfs://.

        And I added a better exception if a host+port string, that is given in the send method will not be splittable by ":".

        I wish the person how want to test this the best of luck

        Show
        Thomas Jungblut added a comment - I have fixed PiEstimator now, it works in both modes. Additionally I have added some outputs when starting a cluster in local mode, because I had some mailing list fun the last days. If the apps determine that there is "local" in "bsp.master.adress" then the application exits and prints that it doesn't need to be started. This will now prevent the questions: "my error log shows, local is no valid hostname". Although I think this is a pretty clear exception >_>. Anyways, you can now submit local jobs via the shell ./hama jar <jar.jar> test for example. And I added testcases for the local mode and fixed a bit with the name handling in our cluster status. There were a hell lot of other problems, for example a local running BSP that could not work on DFS because the filename wasn't qualified with hdfs://. And I added a better exception if a host+port string, that is given in the send method will not be splittable by ":". I wish the person how want to test this the best of luck
        Hide
        Hudson added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12494849/HAMA-437.patch
        against trunk revision 1171537.

        @author +1. The patch does not contain any @author tags.

        tests included +1. The patch appears to include 10 new or modified tests.

        core tests -1. The patch failed core unit tests.

        Changes : http://builds.apache.org/hudson/job/Hama-Patch/356/changes/
        Console output: http://builds.apache.org/hudson/job/Hama-Patch/356/console

        This message is automatically generated.

        Show
        Hudson added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12494849/HAMA-437.patch against trunk revision 1171537. @author +1. The patch does not contain any @author tags. tests included +1. The patch appears to include 10 new or modified tests. core tests -1. The patch failed core unit tests. Changes : http://builds.apache.org/hudson/job/Hama-Patch/356/changes/ Console output: http://builds.apache.org/hudson/job/Hama-Patch/356/console This message is automatically generated.
        Hide
        Thomas Jungblut added a comment -
        11/09/16 15:11:57 INFO bsp.TestLocalBSPRunner: Starting job...
        
        Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.023 sec <<< FAILURE!
        
        

        This test seems not to be jenkins compatible. It works on my laptop.
        Gladly there is so much information in these two lines.

        Show
        Thomas Jungblut added a comment - 11/09/16 15:11:57 INFO bsp.TestLocalBSPRunner: Starting job... Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.023 sec <<< FAILURE! This test seems not to be jenkins compatible. It works on my laptop. Gladly there is so much information in these two lines.
        Hide
        Thomas Jungblut added a comment -

        Fix the localrunner testcase

        Show
        Thomas Jungblut added a comment - Fix the localrunner testcase
        Hide
        Hudson added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12495198/HAMA-437_v2.patch
        against trunk revision 1171537.

        @author +1. The patch does not contain any @author tags.

        tests included +1. The patch appears to include 3 new or modified tests.

        core tests -1. The patch failed core unit tests.

        Changes : http://builds.apache.org/hudson/job/Hama-Patch/357/changes/
        Console output: http://builds.apache.org/hudson/job/Hama-Patch/357/console

        This message is automatically generated.

        Show
        Hudson added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12495198/HAMA-437_v2.patch against trunk revision 1171537. @author +1. The patch does not contain any @author tags. tests included +1. The patch appears to include 3 new or modified tests. core tests -1. The patch failed core unit tests. Changes : http://builds.apache.org/hudson/job/Hama-Patch/357/changes/ Console output: http://builds.apache.org/hudson/job/Hama-Patch/357/console This message is automatically generated.
        Hide
        Thomas Jungblut added a comment -

        This time the Checkpointer failed.
        Seems that Jenkins is not working properly.

        Would someone please apply the test on his trunk and run the testcases?
        For me, they are working.

        Show
        Thomas Jungblut added a comment - This time the Checkpointer failed. Seems that Jenkins is not working properly. Would someone please apply the test on his trunk and run the testcases? For me, they are working.
        Hide
        Edward J. Yoon added a comment -

        I manually re-scheduled this again and worked this time.

        +1

        Show
        Edward J. Yoon added a comment - I manually re-scheduled this again and worked this time. +1
        Hide
        Hudson added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12495198/HAMA-437_v2.patch
        against trunk revision 1171537.

        @author +1. The patch does not contain any @author tags.

        tests included +1. The patch appears to include 3 new or modified tests.

        core tests +1. The patch passed core unit tests.

        Changes : http://builds.apache.org/hudson/job/Hama-Patch/358/changes/
        Console output: http://builds.apache.org/hudson/job/Hama-Patch/358/console

        This message is automatically generated.

        Show
        Hudson added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12495198/HAMA-437_v2.patch against trunk revision 1171537. @author +1. The patch does not contain any @author tags. tests included +1. The patch appears to include 3 new or modified tests. core tests +1. The patch passed core unit tests. Changes : http://builds.apache.org/hudson/job/Hama-Patch/358/changes/ Console output: http://builds.apache.org/hudson/job/Hama-Patch/358/console This message is automatically generated.
        Hide
        Edward J. Yoon added a comment -

        Before commit this, Please see http://wiki.apache.org/hama/HowToCommit

        Keep up the great work, Thomas!

        Show
        Edward J. Yoon added a comment - Before commit this, Please see http://wiki.apache.org/hama/HowToCommit Keep up the great work, Thomas!
        Hide
        Thomas Jungblut added a comment -

        Thanks! I hope our users will now better distinct between the modes.

        Before commit this, Please see http://wiki.apache.org/hama/HowToCommit

        I already setuped the SVN client, but I'm missing the rights on our SVN repo.
        I mailed Tommaso, he'll take care of the rights.
        Feel free to commit this as long as I don't own the rights.

        Show
        Thomas Jungblut added a comment - Thanks! I hope our users will now better distinct between the modes. Before commit this, Please see http://wiki.apache.org/hama/HowToCommit I already setuped the SVN client, but I'm missing the rights on our SVN repo. I mailed Tommaso, he'll take care of the rights. Feel free to commit this as long as I don't own the rights.
        Hide
        Thomas Jungblut added a comment -

        Just committed this.
        Thanks to all of you!

        Show
        Thomas Jungblut added a comment - Just committed this. Thanks to all of you!
        Hide
        Hudson added a comment -

        Integrated in Hama-Nightly #316 (See https://builds.apache.org/job/Hama-Nightly/316/)
        HAMA-437 - fix for local mode, testcases and examples.

        tjungblut :
        Files :

        • /incubator/hama/trunk/CHANGES.txt
        • /incubator/hama/trunk/core/src/main/java/org/apache/hama/bsp/BSPJob.java
        • /incubator/hama/trunk/core/src/main/java/org/apache/hama/bsp/BSPJobClient.java
        • /incubator/hama/trunk/core/src/main/java/org/apache/hama/bsp/BSPMaster.java
        • /incubator/hama/trunk/core/src/main/java/org/apache/hama/bsp/BSPPeer.java
        • /incubator/hama/trunk/core/src/main/java/org/apache/hama/bsp/GroomServer.java
        • /incubator/hama/trunk/core/src/main/java/org/apache/hama/bsp/LocalBSPRunner.java
        • /incubator/hama/trunk/core/src/main/java/org/apache/hama/zookeeper/QuorumPeer.java
        • /incubator/hama/trunk/core/src/test/java/org/apache/hama/bsp/TestBSPMasterGroomServer.java
        • /incubator/hama/trunk/examples/src/main/java/org/apache/hama/examples/PiEstimator.java
        • /incubator/hama/trunk/examples/src/main/java/org/apache/hama/examples/graph/PageRank.java
        • /incubator/hama/trunk/examples/src/main/java/org/apache/hama/examples/graph/PageRankBase.java
        • /incubator/hama/trunk/examples/src/main/java/org/apache/hama/examples/graph/ShortestPaths.java
        • /incubator/hama/trunk/examples/src/main/java/org/apache/hama/examples/graph/ShortestPathsBase.java
        Show
        Hudson added a comment - Integrated in Hama-Nightly #316 (See https://builds.apache.org/job/Hama-Nightly/316/ ) HAMA-437 - fix for local mode, testcases and examples. tjungblut : Files : /incubator/hama/trunk/CHANGES.txt /incubator/hama/trunk/core/src/main/java/org/apache/hama/bsp/BSPJob.java /incubator/hama/trunk/core/src/main/java/org/apache/hama/bsp/BSPJobClient.java /incubator/hama/trunk/core/src/main/java/org/apache/hama/bsp/BSPMaster.java /incubator/hama/trunk/core/src/main/java/org/apache/hama/bsp/BSPPeer.java /incubator/hama/trunk/core/src/main/java/org/apache/hama/bsp/GroomServer.java /incubator/hama/trunk/core/src/main/java/org/apache/hama/bsp/LocalBSPRunner.java /incubator/hama/trunk/core/src/main/java/org/apache/hama/zookeeper/QuorumPeer.java /incubator/hama/trunk/core/src/test/java/org/apache/hama/bsp/TestBSPMasterGroomServer.java /incubator/hama/trunk/examples/src/main/java/org/apache/hama/examples/PiEstimator.java /incubator/hama/trunk/examples/src/main/java/org/apache/hama/examples/graph/PageRank.java /incubator/hama/trunk/examples/src/main/java/org/apache/hama/examples/graph/PageRankBase.java /incubator/hama/trunk/examples/src/main/java/org/apache/hama/examples/graph/ShortestPaths.java /incubator/hama/trunk/examples/src/main/java/org/apache/hama/examples/graph/ShortestPathsBase.java

          People

          • Assignee:
            Thomas Jungblut
            Reporter:
            Thomas Jungblut
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development