Details

    • Type: New Feature New Feature
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.4.0
    • Fix Version/s: 0.6.0
    • Component/s: None
    • Labels:
      None

      Description

      This issue adds Hama as a Service to Whirr.

      I just wrote some code today but haven't run the integration tests yet.

      1. HAMA.patch
        0.3 kB
        Edward J. Yoon
      2. WHIRR-313_v01.patch
        43 kB
        Edward J. Yoon
      3. WHIRR-313_v02.patch
        44 kB
        Edward J. Yoon
      4. WHIRR-313_v03.patch
        46 kB
        Edward J. Yoon
      5. WHIRR-313_v04.patch
        46 kB
        Edward J. Yoon
      6. WHIRR-313_v05.patch
        45 kB
        Edward J. Yoon

        Activity

        Hide
        Edward J. Yoon added a comment -

        Attach my patch,and here's 'hama service' tar file - http://people.apache.org/~edwardyoon/hama.tar.gz

        Show
        Edward J. Yoon added a comment - Attach my patch,and here's 'hama service' tar file - http://people.apache.org/~edwardyoon/hama.tar.gz
        Hide
        Andrei Savu added a comment -

        Let's move this to 0.6.0 so that we can perform more testing. We are planning to release 0.5.0 soon.

        Show
        Andrei Savu added a comment - Let's move this to 0.6.0 so that we can perform more testing. We are planning to release 0.5.0 soon.
        Hide
        Andrei Savu added a comment -

        Looks good! Unfortunately the integration tests are not working for me.

        Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 279.201 sec <<< FAILURE!
        test(org.apache.whirr.service.hama.integration.HamaServiceTest)  Time elapsed: 11.504 sec  <<< ERROR!
        java.io.IOException: Call to localhost/127.0.0.1:40000 failed on local exception: java.net.SocketException: Malformed reply from SOCKS server
                at org.apache.hadoop.ipc.Client.wrapException(Client.java:775)
                at org.apache.hadoop.ipc.Client.call(Client.java:743)
                at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
                at $Proxy81.getProtocolVersion(Unknown Source)
                at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:359)
                at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:346)
                at org.apache.hama.bsp.BSPJobClient.init(BSPJobClient.java:200)
                at org.apache.hama.bsp.BSPJobClient.<init>(BSPJobClient.java:193)
                at org.apache.hama.bsp.BSPJob.<init>(BSPJob.java:48)
                at org.apache.hama.bsp.BSPJob.<init>(BSPJob.java:61)
                at org.apache.whirr.service.hama.integration.HamaServiceTest.test(HamaServiceTest.java:47)
        

        I've noticed some logging related issues that need to be addressed:

        SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
        SLF4J: Defaulting to no-operation (NOP) logger implementation
        SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
        

        Also we should add whirr-hama to cli/pom.xml and provide a recipe.

        Later we could also create an example that shows how to use Hama with Whirr (WHIRR-28).

        Show
        Andrei Savu added a comment - Looks good! Unfortunately the integration tests are not working for me. Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 279.201 sec <<< FAILURE! test(org.apache.whirr.service.hama.integration.HamaServiceTest) Time elapsed: 11.504 sec <<< ERROR! java.io.IOException: Call to localhost/127.0.0.1:40000 failed on local exception: java.net.SocketException: Malformed reply from SOCKS server at org.apache.hadoop.ipc.Client.wrapException(Client.java:775) at org.apache.hadoop.ipc.Client.call(Client.java:743) at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220) at $Proxy81.getProtocolVersion(Unknown Source) at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:359) at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:346) at org.apache.hama.bsp.BSPJobClient.init(BSPJobClient.java:200) at org.apache.hama.bsp.BSPJobClient.<init>(BSPJobClient.java:193) at org.apache.hama.bsp.BSPJob.<init>(BSPJob.java:48) at org.apache.hama.bsp.BSPJob.<init>(BSPJob.java:61) at org.apache.whirr.service.hama.integration.HamaServiceTest.test(HamaServiceTest.java:47) I've noticed some logging related issues that need to be addressed: SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: Defaulting to no-operation (NOP) logger implementation SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. Also we should add whirr-hama to cli/pom.xml and provide a recipe. Later we could also create an example that shows how to use Hama with Whirr ( WHIRR-28 ).
        Hide
        Edward J. Yoon added a comment -
        Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 504.726 sec <<< FAILURE!
        org.apache.whirr.service.hama.integration.HamaServiceTest  Time elapsed: 0 sec  <<< ERROR!
        java.io.IOException: Call to ec2-50-19-129-155.compute-1.amazonaws.com/50.19.129.155:40000 failed on local exception: java.net.SocketException: Malformed reply from SOCKS server
        	at org.apache.hadoop.ipc.Client.wrapException(Client.java:775)
        	at org.apache.hadoop.ipc.Client.call(Client.java:743)
        

        Hmm, I'm trying to fix above problem. Is it related with firewall?

        Show
        Edward J. Yoon added a comment - Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 504.726 sec <<< FAILURE! org.apache.whirr.service.hama.integration.HamaServiceTest Time elapsed: 0 sec <<< ERROR! java.io.IOException: Call to ec2-50-19-129-155.compute-1.amazonaws.com/50.19.129.155:40000 failed on local exception: java.net.SocketException: Malformed reply from SOCKS server at org.apache.hadoop.ipc.Client.wrapException(Client.java:775) at org.apache.hadoop.ipc.Client.call(Client.java:743) Hmm, I'm trying to fix above problem. Is it related with firewall?
        Hide
        Andrei Savu added a comment -

        It shouldn't be related to the firewall settings. I would try to start a cluster outside the integration tests and inspect everything by hand (including the SOCKS proxy script).

        Show
        Andrei Savu added a comment - It shouldn't be related to the firewall settings. I would try to start a cluster outside the integration tests and inspect everything by hand (including the SOCKS proxy script).
        Hide
        Edward J. Yoon added a comment -

        Oh, i see what's problem now.

        (attaching my progress before leaving the office.)

        Show
        Edward J. Yoon added a comment - Oh, i see what's problem now. (attaching my progress before leaving the office.)
        Hide
        Edward J. Yoon added a comment -
        
        2011-05-30 20:20:29,971 INFO  [org.apache.whirr.service.hadoop.HadoopNameNodeClusterActionHandler] (main) Namenode web UI available at http://ec2-50-19-68-230.compute-1.amazonaws.com:50070
        2011-05-30 20:20:29,971 INFO  [org.apache.whirr.service.hadoop.HadoopNameNodeClusterActionHandler] (main) Jobtracker web UI available at http://ec2-50-19-68-230.compute-1.amazonaws.com:50030
        2011-05-30 20:20:29,973 INFO  [org.apache.whirr.service.hadoop.HadoopNameNodeClusterActionHandler] (main) Wrote Hadoop site file /home/edward/.whirr/hamaclustertest/hadoop-site.xml
        2011-05-30 20:20:29,975 INFO  [org.apache.whirr.service.hadoop.HadoopNameNodeClusterActionHandler] (main) Wrote Hadoop proxy script /home/edward/.whirr/hamaclustertest/hadoop-proxy.sh
        2011-05-30 20:20:29,975 INFO  [org.apache.whirr.service.hama.HamaMasterClusterActionHandler] (main) Completed configuration of hamaclustertest
        2011-05-30 20:20:29,976 INFO  [org.apache.whirr.service.hama.HamaMasterClusterActionHandler] (main) Wrote Hama site file /home/edward/.whirr/hamaclustertest/hama-site.xml
        2011-05-30 20:20:29,977 INFO  [org.apache.whirr.service.hama.HamaMasterClusterActionHandler] (main) Wrote Hama proxy script /home/edward/.whirr/hamaclustertest/hama-proxy.sh
        2011-05-30 20:20:30,551 INFO  [org.apache.whirr.service.FileClusterStateStore] (main) Wrote instances file /home/edward/.whirr/hamaclustertest/instances
        2011-05-30 20:20:31,748 INFO  [org.apache.whirr.service.hama.integration.HamaServiceController] (main) 1 groomservers reported in. Continuing.
        Warning: Permanently added 'ec2-50-19-68-230.compute-1.amazonaws.com,50.19.68.230' (RSA) to the list of known hosts.
        2011-05-30 20:20:46,053 INFO  [org.apache.whirr.service.hama.integration.HamaServiceController] (main) Shutting down cluster...
        2011-05-30 20:20:46,054 INFO  [org.apache.whirr.actions.DestroyClusterAction] (main) Destroying hamaclustertest cluster
        2011-05-30 20:21:13,880 INFO  [org.apache.whirr.actions.DestroyClusterAction] (main) Cluster hamaclustertest destroyed
        Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 407.136 sec
        
        Results :
        
        Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
        
        [WARNING] File encoding has not been set, using platform encoding UTF-8, i.e. build is platform dependent!
        [INFO] 
        [INFO] --- maven-failsafe-plugin:2.5:verify (verify) @ whirr-hama ---
        [INFO] Failsafe report directory: /home/edward/workspace/whirr-trunk/services/hama/target/failsafe-reports
        [WARNING] File encoding has not been set, using platform encoding UTF-8, i.e. build is platform dependent!
        [INFO] ------------------------------------------------------------------------
        [INFO] Reactor Summary:
        [INFO] 
        [INFO] Whirr ............................................. SUCCESS [1.038s]
        [INFO] Apache Whirr Hama ................................. SUCCESS [6:53.110s]
        

        I've succeeded to run a hama cluster in ec2, remote job submission and whirr integration tests passed.

        Show
        Edward J. Yoon added a comment - 2011-05-30 20:20:29,971 INFO [org.apache.whirr.service.hadoop.HadoopNameNodeClusterActionHandler] (main) Namenode web UI available at http: //ec2-50-19-68-230.compute-1.amazonaws.com:50070 2011-05-30 20:20:29,971 INFO [org.apache.whirr.service.hadoop.HadoopNameNodeClusterActionHandler] (main) Jobtracker web UI available at http: //ec2-50-19-68-230.compute-1.amazonaws.com:50030 2011-05-30 20:20:29,973 INFO [org.apache.whirr.service.hadoop.HadoopNameNodeClusterActionHandler] (main) Wrote Hadoop site file /home/edward/.whirr/hamaclustertest/hadoop-site.xml 2011-05-30 20:20:29,975 INFO [org.apache.whirr.service.hadoop.HadoopNameNodeClusterActionHandler] (main) Wrote Hadoop proxy script /home/edward/.whirr/hamaclustertest/hadoop-proxy.sh 2011-05-30 20:20:29,975 INFO [org.apache.whirr.service.hama.HamaMasterClusterActionHandler] (main) Completed configuration of hamaclustertest 2011-05-30 20:20:29,976 INFO [org.apache.whirr.service.hama.HamaMasterClusterActionHandler] (main) Wrote Hama site file /home/edward/.whirr/hamaclustertest/hama-site.xml 2011-05-30 20:20:29,977 INFO [org.apache.whirr.service.hama.HamaMasterClusterActionHandler] (main) Wrote Hama proxy script /home/edward/.whirr/hamaclustertest/hama-proxy.sh 2011-05-30 20:20:30,551 INFO [org.apache.whirr.service.FileClusterStateStore] (main) Wrote instances file /home/edward/.whirr/hamaclustertest/instances 2011-05-30 20:20:31,748 INFO [org.apache.whirr.service.hama.integration.HamaServiceController] (main) 1 groomservers reported in. Continuing. Warning: Permanently added 'ec2-50-19-68-230.compute-1.amazonaws.com,50.19.68.230' (RSA) to the list of known hosts. 2011-05-30 20:20:46,053 INFO [org.apache.whirr.service.hama.integration.HamaServiceController] (main) Shutting down cluster... 2011-05-30 20:20:46,054 INFO [org.apache.whirr.actions.DestroyClusterAction] (main) Destroying hamaclustertest cluster 2011-05-30 20:21:13,880 INFO [org.apache.whirr.actions.DestroyClusterAction] (main) Cluster hamaclustertest destroyed Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 407.136 sec Results : Tests run: 1, Failures: 0, Errors: 0, Skipped: 0 [WARNING] File encoding has not been set, using platform encoding UTF-8, i.e. build is platform dependent! [INFO] [INFO] --- maven-failsafe-plugin:2.5:verify (verify) @ whirr-hama --- [INFO] Failsafe report directory: /home/edward/workspace/whirr-trunk/services/hama/target/failsafe-reports [WARNING] File encoding has not been set, using platform encoding UTF-8, i.e. build is platform dependent! [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary: [INFO] [INFO] Whirr ............................................. SUCCESS [1.038s] [INFO] Apache Whirr Hama ................................. SUCCESS [6:53.110s] I've succeeded to run a hama cluster in ec2, remote job submission and whirr integration tests passed.
        Hide
        Edward J. Yoon added a comment -

        Could anyone review this patch?

        Show
        Edward J. Yoon added a comment - Could anyone review this patch?
        Hide
        Andrei Savu added a comment -

        I will take a look now.

        Show
        Andrei Savu added a comment - I will take a look now.
        Hide
        Andrei Savu added a comment -

        Looks good! Integration tests pass on ec2 and cloudservers-us. When are you guys planning to have an official release so that we can get rid of the people.apache.org tarball url?

        Typo in recipes/hama-ec2.properties whirr.hbase.tarball.url should probably be whirr.hama.tarball.url

        Show
        Andrei Savu added a comment - Looks good! Integration tests pass on ec2 and cloudservers-us. When are you guys planning to have an official release so that we can get rid of the people.apache.org tarball url? Typo in recipes/hama-ec2.properties whirr.hbase.tarball.url should probably be whirr.hama.tarball.url
        Hide
        Tommaso Teofili added a comment -

        When are you guys planning to have an official release so that we can get rid of the people.apache.org tarball url?

        the vote thread is open on general@incubator.a.o see http://markmail.org/thread/iafalrnyawegtf22 - we just need one more binding +1

        Show
        Tommaso Teofili added a comment - When are you guys planning to have an official release so that we can get rid of the people.apache.org tarball url? the vote thread is open on general@incubator.a.o see http://markmail.org/thread/iafalrnyawegtf22 - we just need one more binding +1
        Hide
        Andrei Savu added a comment -

        the vote thread is open on general@incubator.a.o

        I suggest we wait for an official release before committing. Tom what do you think?

        Also I've forgot to say that you should split the install / configure scripts as suggested in WHRR-266.

        Show
        Andrei Savu added a comment - the vote thread is open on general@incubator.a.o I suggest we wait for an official release before committing. Tom what do you think? Also I've forgot to say that you should split the install / configure scripts as suggested in WHRR-266.
        Hide
        Edward J. Yoon added a comment -

        I have a question for you. Once we create a cluster using whirr, we can't increase the size of cluster without re-creation?

        To shutdown a cluster by one command line, the slave server list (maybe changeable?) should be maintained.

        Show
        Edward J. Yoon added a comment - I have a question for you. Once we create a cluster using whirr, we can't increase the size of cluster without re-creation? To shutdown a cluster by one command line, the slave server list (maybe changeable?) should be maintained.
        Hide
        Andrei Savu added a comment -

        Once we create a cluster using whirr, we can't increase the size of cluster without re-creation?

        Not yet. We are tracking progress on implementing that feature in WHIRR-214 It shouldn't be that hard to do it with a small downtime after we finish WHIRR-266 (this also applies to new services).

        Show
        Andrei Savu added a comment - Once we create a cluster using whirr, we can't increase the size of cluster without re-creation? Not yet. We are tracking progress on implementing that feature in WHIRR-214 It shouldn't be that hard to do it with a small downtime after we finish WHIRR-266 (this also applies to new services).
        Hide
        Edward J. Yoon added a comment -

        Attach new patch. (integration test passed)

        • Fixed typo
        • Added start_hama.sh
        Show
        Edward J. Yoon added a comment - Attach new patch. (integration test passed) Fixed typo Added start_hama.sh
        Hide
        Edward J. Yoon added a comment -
        • Removed people.apache.org tarball url and lib directory.

        All tests passed.

        Show
        Edward J. Yoon added a comment - Removed people.apache.org tarball url and lib directory. All tests passed.
        Hide
        Andrei Savu added a comment -

        +1 Looks good to me. I will wait for one more +1 before committing.

        Show
        Andrei Savu added a comment - +1 Looks good to me. I will wait for one more +1 before committing.
        Hide
        Tom White added a comment -

        Generally looks good. Thanks for the contribution, Edward. Here are a few comments I had after reading through the patch:

        • It's important that the Hama library dependency has test scope, since Whirr shouldn't have a dependency on any particular version - it's only used for the integration test. This applies to hama-examples too.
        • hama.version is defined in pom.xml, but not used.
        • beforeBootstrap() can be pulled up into the abstract cluster handler.
        Show
        Tom White added a comment - Generally looks good. Thanks for the contribution, Edward. Here are a few comments I had after reading through the patch: It's important that the Hama library dependency has test scope, since Whirr shouldn't have a dependency on any particular version - it's only used for the integration test. This applies to hama-examples too. hama.version is defined in pom.xml, but not used. beforeBootstrap() can be pulled up into the abstract cluster handler.
        Hide
        Edward J. Yoon added a comment -

        Hi, thanks for your review.

        Here's my new patch.

        Show
        Edward J. Yoon added a comment - Hi, thanks for your review. Here's my new patch.
        Hide
        Tom White added a comment -

        I've just committed this. Thanks, Edward!

        Show
        Tom White added a comment - I've just committed this. Thanks, Edward!

          People

          • Assignee:
            Edward J. Yoon
            Reporter:
            Edward J. Yoon
          • Votes:
            3 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development