Uploaded image for project: 'Apache Whirr (retired)'
  1. Apache Whirr (retired)
  2. WHIRR-330

BYON doesn't work with HadoopConfigurationBuilder

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.5.0
    • Fix Version/s: 0.6.0
    • Component/s: core, service/hadoop
    • Labels:
      None
    • Environment:

      CentOS 5.5

      Description

      I created a hadoop-byon.properties file in recipes similar to recipes/zookeeper-byon.properties, and I customized recipes/nodes-byon.yaml for my machines, then I ran:

      bin/whirr launch-cluster --config recipes/hadoop-byon.properties
      

      I see the following error:

      Exception in thread "main" java.lang.NullPointerException
              at org.apache.whirr.service.hadoop.HadoopConfigurationBuilder.buildMapReduceConfiguration(HadoopConfigurationBuilder.java:106)
              at org.apache.whirr.service.hadoop.HadoopConfigurationBuilder.buildMapReduce(HadoopConfigurationBuilder.java:71)
              at org.apache.whirr.service.hadoop.HadoopNameNodeClusterActionHandler.beforeConfigure(HadoopNameNodeClusterActionHandler.java:80)
              at org.apache.whirr.service.ClusterActionHandlerSupport.beforeAction(ClusterActionHandlerSupport.java:48)
              at org.apache.whirr.actions.ScriptBasedClusterAction.execute(ScriptBasedClusterAction.java:75)
              at org.apache.whirr.ByonClusterController.launchCluster(ByonClusterController.java:54)
              at org.apache.whirr.cli.command.LaunchClusterCommand.run(LaunchClusterCommand.java:61)
              at org.apache.whirr.cli.Main.run(Main.java:68)
              at org.apache.whirr.cli.Main.main(Main.java:106)
      

      The code looks like this:

      Hardware hardware = Iterables.getFirst(taskTrackers, null)
              .getNodeMetadata().getHardware();
      int coresPerNode = 0;
      for (Processor processor : hardware.getProcessors()) {
        coresPerNode += processor.getCores();
      }
      

      The problem is that ByonClusterAction.java passes a null NodeMetadata when it creates a new Instance:

      public Instance apply(NodeMetadata node) {
        String publicIp = Iterables.get(node.getPublicAddresses(), 0);
        return new Instance(credentials, roles, publicIp, publicIp, node.getId(), null);
      }
      

      I changed null to node in the above code and it only helps a little because then a NullPointerException is thrown at HadoopConfigurationBuilder.java:109 because the NodeMetadata has a null Hardware attribute. It seems like one of two things could happen:
      (1) recipes/nodes-byon.yaml could be enhanced to allow specification of hardware, including processors and cores per processor (and the first NullPointerException at line 106 would still need to be fixed)
      (2) The machines could be queried at runtime using /proc/cpuinfo, but that would probably need to happen in jclouds.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                savu.andrei Andrei Savu
                Reporter:
                johnamos John Amos
              • Votes:
                0 Vote for this issue
                Watchers:
                0 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: