Whirr
  1. Whirr
  2. WHIRR-571

HadoopConfigurationBuilder assumes that a Hadoop cluster has both a namenode and a jobtracker

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Trivial Trivial
    • Resolution: Fixed
    • Affects Version/s: 0.7.1
    • Fix Version/s: 0.7.2
    • Component/s: service/hadoop
    • Labels:
      None

      Description

      I removed hadoop-jobtracker and hadoop-tasktracker from whirr.instance-templates in hbase-ec2.properties.
      Then I got the following error when trying to start a cluster.

      Am I doing anything wrong by removing the tracker from instance templates?
      Or is this an issue or not?
      If it is not an issue, am I be able to start hbase without starting the jobtracker and tasktracker? What is the right way to do it?

      Authorizing firewall ingress to [us-east-1/i-98c4f0fc] on ports [50070] for [128.83.158.52/32]
      Authorizing firewall ingress to [us-east-1/i-98c4f0fc] on ports [8020, 8021] for [174.129.136.180/32]
      Unable to start the cluster. Terminating all nodes.
      java.util.NoSuchElementException
      at com.google.common.collect.AbstractIterator.next(AbstractIterator.java:152)
      at com.google.common.collect.Iterators.getOnlyElement(Iterators.java:293)
      at com.google.common.collect.Iterables.getOnlyElement(Iterables.java:265)
      at org.apache.whirr.Cluster.getInstanceMatching(Cluster.java:171)
      at org.apache.whirr.service.hadoop.HadoopConfigurationBuilder.buildMapReduceConfiguration(HadoopConfigurationBuilder.java:135)
      at org.apache.whirr.service.hadoop.HadoopConfigurationBuilder.buildMapReduce(HadoopConfigurationBuilder.java:71)
      at org.apache.whirr.service.hadoop.HadoopClusterActionHandler.createHadoopConfigFiles(HadoopClusterActionHandler.java:104)
      at org.apache.whirr.service.hadoop.HadoopClusterActionHandler.beforeConfigure(HadoopClusterActionHandler.java:89)
      at org.apache.whirr.service.ClusterActionHandlerSupport.beforeAction(ClusterActionHandlerSupport.java:53)
      at org.apache.whirr.actions.ScriptBasedClusterAction.execute(ScriptBasedClusterAction.java:100)
      at org.apache.whirr.ClusterController.launchCluster(ClusterController.java:109)
      at org.apache.whirr.cli.command.LaunchClusterCommand.run(LaunchClusterCommand.java:63)
      at org.apache.whirr.cli.Main.run(Main.java:64)
      at org.apache.whirr.cli.Main.main(Main.java:97)
      Unable to load cluster state, assuming it has no running nodes.

      1. whirr.log
        16 kB
        Zuocheng Ren
      2. exalt-hbase-ec2.properties
        3 kB
        Zuocheng Ren

        Activity

        Hide
        Andrei Savu added a comment -

        Marking as fixed in 0.7.2.

        Show
        Andrei Savu added a comment - Marking as fixed in 0.7.2.
        Hide
        Andrei Savu added a comment -

        I've just tried the same recipe with 0.7.1 and I get the same exception you are seeing. The good news is that this issue is fixed in branch 0.7 and it's going to be part of the 0.7.2 release.

        Show
        Andrei Savu added a comment - I've just tried the same recipe with 0.7.1 and I get the same exception you are seeing. The good news is that this issue is fixed in branch 0.7 and it's going to be part of the 0.7.2 release.
        Hide
        Zuocheng Ren added a comment -

        Sure.
        I've attached my recipe to this issue.
        The recipe attached will cause an exception during launching a cluster.
        But if I add hadoop-jobtracker and hadoop-tasktracker, the cluster can start successfully.
        BTW, I didn't make any change to the 0.7.1 release.

        Show
        Zuocheng Ren added a comment - Sure. I've attached my recipe to this issue. The recipe attached will cause an exception during launching a cluster. But if I add hadoop-jobtracker and hadoop-tasktracker, the cluster can start successfully. BTW, I didn't make any change to the 0.7.1 release.
        Hide
        Andrei Savu added a comment -

        HadoopConfigurationBuilder.java was not changed recently. Do you have local modifications or this is the standard release?

        Show
        Andrei Savu added a comment - HadoopConfigurationBuilder.java was not changed recently. Do you have local modifications or this is the standard release?
        Hide
        Andrei Savu added a comment -

        I am unable to reproduce this issue with trunk. I will also check with the 0.7 branch. Can you share your updated recipe? I was able to start a cluster running only HDFS (hadoop-namenode & hadoop-datanode) on aws-ec2.

        Show
        Andrei Savu added a comment - I am unable to reproduce this issue with trunk. I will also check with the 0.7 branch. Can you share your updated recipe? I was able to start a cluster running only HDFS (hadoop-namenode & hadoop-datanode) on aws-ec2.
        Hide
        Zuocheng Ren added a comment -

        As mentioned by Andrei Savu, "it should be easy to fix: catch the NoSuchElementException in
        Cluster.getInstancesMatching and return an empty set."

        Show
        Zuocheng Ren added a comment - As mentioned by Andrei Savu, "it should be easy to fix: catch the NoSuchElementException in Cluster.getInstancesMatching and return an empty set."

          People

          • Assignee:
            Unassigned
            Reporter:
            Zuocheng Ren
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development