Apache Whirr (retired)
  1. Apache Whirr (retired)
  2. WHIRR-191

[CDH] Start other services based on CDH, not just HDFS and MR

    Details

    • Type: New Feature New Feature
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.3.0
    • Fix Version/s: 0.5.0
    • Component/s: service/hadoop
    • Labels:
      None

      Description

      When using the CDH service it currently only starts HDFS and MR, similar to the Apache Hadoop service (the one that uses the tarballs). We do have for example also an Apache ZooKeeper or Apache HBase service, but for CDH we do not. Either add (and change the current service) a CDH Hadoop, CDH HBase and so on or come up with another means of starting up other processes apart from HDFS and MR based on the CDH release.

      1. WHIRR-191.patch
        44 kB
        Tom White
      2. WHIRR-191.patch
        38 kB
        Tom White
      3. WHIRR-191.patch
        26 kB
        Tom White

        Issue Links

          Activity

          Hide
          Tom White added a comment -

          Can we do what we do for HDFS and MR? That is, have separate install and configure scripts, but share the ClusterActionHandlers?

          Show
          Tom White added a comment - Can we do what we do for HDFS and MR? That is, have separate install and configure scripts, but share the ClusterActionHandlers?
          Hide
          Lars George added a comment -

          I would think so, since it is only up on the servers which is different, i.e. they are all CDH based. We need to add a parameter for repo as well.

          Show
          Lars George added a comment - I would think so, since it is only up on the servers which is different, i.e. they are all CDH based. We need to add a parameter for repo as well.
          Hide
          Tom White added a comment -

          Work in progress patch.

          Show
          Tom White added a comment - Work in progress patch.
          Hide
          Tom White added a comment -

          To complete this we need to write the HBase integration test using the Thrift interface, since CDH doesn't package the REST interface. This is related to WHIRR-192.

          Show
          Tom White added a comment - To complete this we need to write the HBase integration test using the Thrift interface, since CDH doesn't package the REST interface. This is related to WHIRR-192 .
          Hide
          Tom White added a comment -

          With help from Lars I updated the test to use the Thrift server, but I'm having problems getting a scanner (as a test that HBase is running). The exception I get is:

          org.apache.thrift.TApplicationException: Internal error processing scannerOpen
          	at org.apache.thrift.TApplicationException.read(TApplicationException.java:107)
          	at org.apache.hadoop.hbase.thrift.generated.Hbase$Client.recv_scannerOpen(Hbase.java:1473)
          	at org.apache.hadoop.hbase.thrift.generated.Hbase$Client.scannerOpen(Hbase.java:1454)
          	at org.apache.whirr.service.hbase.integration.HBaseServiceController.getScanner(HBaseServiceController.java:154)
          	at org.apache.whirr.service.hbase.integration.HBaseServiceController.waitForMaster(HBaseServiceController.java:137)
          	at org.apache.whirr.service.hbase.integration.HBaseServiceController.startup(HBaseServiceController.java:102)
          	at org.apache.whirr.service.hbase.integration.HBaseServiceController.ensureClusterRunning(HBaseServiceController.java:78)
          	at org.apache.whirr.service.hbase.integration.HBaseServiceTest.setUp(HBaseServiceTest.java:57)
          	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
          	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          	at java.lang.reflect.Method.invoke(Method.java:597)
          	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
          	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
          	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
          	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:27)
          	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
          	at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
          	at org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:59)
          	at org.apache.maven.surefire.suite.AbstractDirectoryTestSuite.executeTestSet(AbstractDirectoryTestSuite.java:115)
          	at org.apache.maven.surefire.suite.AbstractDirectoryTestSuite.execute(AbstractDirectoryTestSuite.java:102)
          	at org.apache.maven.surefire.Surefire.run(Surefire.java:180)
          	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
          	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          	at java.lang.reflect.Method.invoke(Method.java:597)
          	at org.apache.maven.surefire.booter.SurefireBooter.runSuitesInProcess(SurefireBooter.java:350)
          	at org.apache.maven.surefire.booter.SurefireBooter.main(SurefireBooter.java:1021)
          

          Any ideas about how to make progress from here?

          Show
          Tom White added a comment - With help from Lars I updated the test to use the Thrift server, but I'm having problems getting a scanner (as a test that HBase is running). The exception I get is: org.apache.thrift.TApplicationException: Internal error processing scannerOpen at org.apache.thrift.TApplicationException.read(TApplicationException.java:107) at org.apache.hadoop.hbase.thrift.generated.Hbase$Client.recv_scannerOpen(Hbase.java:1473) at org.apache.hadoop.hbase.thrift.generated.Hbase$Client.scannerOpen(Hbase.java:1454) at org.apache.whirr.service.hbase.integration.HBaseServiceController.getScanner(HBaseServiceController.java:154) at org.apache.whirr.service.hbase.integration.HBaseServiceController.waitForMaster(HBaseServiceController.java:137) at org.apache.whirr.service.hbase.integration.HBaseServiceController.startup(HBaseServiceController.java:102) at org.apache.whirr.service.hbase.integration.HBaseServiceController.ensureClusterRunning(HBaseServiceController.java:78) at org.apache.whirr.service.hbase.integration.HBaseServiceTest.setUp(HBaseServiceTest.java:57) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:27) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31) at org.junit.runners.ParentRunner.run(ParentRunner.java:236) at org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:59) at org.apache.maven.surefire.suite.AbstractDirectoryTestSuite.executeTestSet(AbstractDirectoryTestSuite.java:115) at org.apache.maven.surefire.suite.AbstractDirectoryTestSuite.execute(AbstractDirectoryTestSuite.java:102) at org.apache.maven.surefire.Surefire.run(Surefire.java:180) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.maven.surefire.booter.SurefireBooter.runSuitesInProcess(SurefireBooter.java:350) at org.apache.maven.surefire.booter.SurefireBooter.main(SurefireBooter.java:1021) Any ideas about how to make progress from here?
          Hide
          Tom White added a comment -

          This is now working (the problem was with passing nulls to get the scanner). I've tested all the Hadoop, HBase, ZooKeeper services and they pass on EC2 so this should be ready to go.

          Show
          Tom White added a comment - This is now working (the problem was with passing nulls to get the scanner). I've tested all the Hadoop, HBase, ZooKeeper services and they pass on EC2 so this should be ready to go.
          Hide
          Lars George added a comment -

          I looked through the patch, looks good to me. If the tests pass then +1 from me.

          Show
          Lars George added a comment - I looked through the patch, looks good to me. If the tests pass then +1 from me.
          Hide
          Tom White added a comment -

          I've just committed this.

          Show
          Tom White added a comment - I've just committed this.

            People

            • Assignee:
              Tom White
              Reporter:
              Lars George
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development