Whirr
  1. Whirr
  2. WHIRR-330

BYON doesn't work with HadoopConfigurationBuilder

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.5.0
    • Fix Version/s: 0.6.0
    • Component/s: core, service/hadoop
    • Labels:
      None
    • Environment:

      CentOS 5.5

      Description

      I created a hadoop-byon.properties file in recipes similar to recipes/zookeeper-byon.properties, and I customized recipes/nodes-byon.yaml for my machines, then I ran:

      bin/whirr launch-cluster --config recipes/hadoop-byon.properties
      

      I see the following error:

      Exception in thread "main" java.lang.NullPointerException
              at org.apache.whirr.service.hadoop.HadoopConfigurationBuilder.buildMapReduceConfiguration(HadoopConfigurationBuilder.java:106)
              at org.apache.whirr.service.hadoop.HadoopConfigurationBuilder.buildMapReduce(HadoopConfigurationBuilder.java:71)
              at org.apache.whirr.service.hadoop.HadoopNameNodeClusterActionHandler.beforeConfigure(HadoopNameNodeClusterActionHandler.java:80)
              at org.apache.whirr.service.ClusterActionHandlerSupport.beforeAction(ClusterActionHandlerSupport.java:48)
              at org.apache.whirr.actions.ScriptBasedClusterAction.execute(ScriptBasedClusterAction.java:75)
              at org.apache.whirr.ByonClusterController.launchCluster(ByonClusterController.java:54)
              at org.apache.whirr.cli.command.LaunchClusterCommand.run(LaunchClusterCommand.java:61)
              at org.apache.whirr.cli.Main.run(Main.java:68)
              at org.apache.whirr.cli.Main.main(Main.java:106)
      

      The code looks like this:

      Hardware hardware = Iterables.getFirst(taskTrackers, null)
              .getNodeMetadata().getHardware();
      int coresPerNode = 0;
      for (Processor processor : hardware.getProcessors()) {
        coresPerNode += processor.getCores();
      }
      

      The problem is that ByonClusterAction.java passes a null NodeMetadata when it creates a new Instance:

      public Instance apply(NodeMetadata node) {
        String publicIp = Iterables.get(node.getPublicAddresses(), 0);
        return new Instance(credentials, roles, publicIp, publicIp, node.getId(), null);
      }
      

      I changed null to node in the above code and it only helps a little because then a NullPointerException is thrown at HadoopConfigurationBuilder.java:109 because the NodeMetadata has a null Hardware attribute. It seems like one of two things could happen:
      (1) recipes/nodes-byon.yaml could be enhanced to allow specification of hardware, including processors and cores per processor (and the first NullPointerException at line 106 would still need to be fixed)
      (2) The machines could be queried at runtime using /proc/cpuinfo, but that would probably need to happen in jclouds.

        Issue Links

          Activity

          John Amos created issue -
          John Amos made changes -
          Field Original Value New Value
          Link This issue is related to WHIRR-285 [ WHIRR-285 ]
          John Amos made changes -
          Link This issue is related to WHIRR-299 [ WHIRR-299 ]
          John Amos made changes -
          Description I created a hadoop-byon.properties file in recipes similar to recipes/zookeeper-byon.properties, and I customized recipes/nodes-byon.yaml for my machines, then I ran:

          {noformat}
          bin/whirr launch-cluster --config recipes/hadoop-byon.properties
          {noformat}

          I see the following error:

          {noformat}
          Exception in thread "main" java.lang.NullPointerException
                  at org.apache.whirr.service.hadoop.HadoopConfigurationBuilder.buildMapReduceConfiguration(HadoopConfigurationBuilder.java:106)
                  at org.apache.whirr.service.hadoop.HadoopConfigurationBuilder.buildMapReduce(HadoopConfigurationBuilder.java:71)
                  at org.apache.whirr.service.hadoop.HadoopNameNodeClusterActionHandler.beforeConfigure(HadoopNameNodeClusterActionHandler.java:80)
                  at org.apache.whirr.service.ClusterActionHandlerSupport.beforeAction(ClusterActionHandlerSupport.java:48)
                  at org.apache.whirr.actions.ScriptBasedClusterAction.execute(ScriptBasedClusterAction.java:75)
                  at org.apache.whirr.ByonClusterController.launchCluster(ByonClusterController.java:54)
                  at org.apache.whirr.cli.command.LaunchClusterCommand.run(LaunchClusterCommand.java:61)
                  at org.apache.whirr.cli.Main.run(Main.java:68)
                  at org.apache.whirr.cli.Main.main(Main.java:106)
          {noformat}

          The code looks like this:
          {code:java}
                Hardware hardware = Iterables.getFirst(taskTrackers, null)
                  .getNodeMetadata().getHardware();
                int coresPerNode = 0;
                for (Processor processor : hardware.getProcessors()) {
                  coresPerNode += processor.getCores();
                }
          {code}

          The problem is that ByonClusterAction.java passes a null NodeMetadata when it creates a new Instance:
          {code:java}
                public Instance apply(NodeMetadata node) {
                  String publicIp = Iterables.get(node.getPublicAddresses(), 0);
                  return new Instance(credentials, roles, publicIp, publicIp, node.getId(), null);
                }
          {code}

          I changed null to node in the above code and it only helps a little because then a NullPointerException is thrown at HadoopConfigurationBuilder.java:109 because the NodeMetadata has a null Hardware attribute. It seems like one of two things could happen:
          (1) recipes/nodes-byon.yaml could be enhanced to allow specification of hardware, including processors and cores per processor (and the first NullPointerException at line 106 would still need to be fixed)
          (2) The machines could be queried at runtime using /proc/cpuinfo, but that would probably need to happen in jclouds.
          I created a hadoop-byon.properties file in recipes similar to recipes/zookeeper-byon.properties, and I customized recipes/nodes-byon.yaml for my machines, then I ran:

          {noformat}
          bin/whirr launch-cluster --config recipes/hadoop-byon.properties
          {noformat}

          I see the following error:

          {noformat}
          Exception in thread "main" java.lang.NullPointerException
                  at org.apache.whirr.service.hadoop.HadoopConfigurationBuilder.buildMapReduceConfiguration(HadoopConfigurationBuilder.java:106)
                  at org.apache.whirr.service.hadoop.HadoopConfigurationBuilder.buildMapReduce(HadoopConfigurationBuilder.java:71)
                  at org.apache.whirr.service.hadoop.HadoopNameNodeClusterActionHandler.beforeConfigure(HadoopNameNodeClusterActionHandler.java:80)
                  at org.apache.whirr.service.ClusterActionHandlerSupport.beforeAction(ClusterActionHandlerSupport.java:48)
                  at org.apache.whirr.actions.ScriptBasedClusterAction.execute(ScriptBasedClusterAction.java:75)
                  at org.apache.whirr.ByonClusterController.launchCluster(ByonClusterController.java:54)
                  at org.apache.whirr.cli.command.LaunchClusterCommand.run(LaunchClusterCommand.java:61)
                  at org.apache.whirr.cli.Main.run(Main.java:68)
                  at org.apache.whirr.cli.Main.main(Main.java:106)
          {noformat}

          The code looks like this:
          {code:java}
          Hardware hardware = Iterables.getFirst(taskTrackers, null)
                  .getNodeMetadata().getHardware();
          int coresPerNode = 0;
          for (Processor processor : hardware.getProcessors()) {
            coresPerNode += processor.getCores();
          }
          {code}

          The problem is that ByonClusterAction.java passes a null NodeMetadata when it creates a new Instance:
          {code:java}
          public Instance apply(NodeMetadata node) {
            String publicIp = Iterables.get(node.getPublicAddresses(), 0);
            return new Instance(credentials, roles, publicIp, publicIp, node.getId(), null);
          }
          {code}

          I changed null to node in the above code and it only helps a little because then a NullPointerException is thrown at HadoopConfigurationBuilder.java:109 because the NodeMetadata has a null Hardware attribute. It seems like one of two things could happen:
          (1) recipes/nodes-byon.yaml could be enhanced to allow specification of hardware, including processors and cores per processor (and the first NullPointerException at line 106 would still need to be fixed)
          (2) The machines could be queried at runtime using /proc/cpuinfo, but that would probably need to happen in jclouds.
          Component/s core [ 12313574 ]
          Hide
          Tom White added a comment -

          We could also make HadoopConfigurationBuilder more defensive so that it falls back to Hadoop defaults if Hardware is null. This is probably the simplest fix.

          Fix 1 seems like a good thing to do too, but it requires a jclouds change.

          Show
          Tom White added a comment - We could also make HadoopConfigurationBuilder more defensive so that it falls back to Hadoop defaults if Hardware is null. This is probably the simplest fix. Fix 1 seems like a good thing to do too, but it requires a jclouds change.
          Hide
          Andrei Savu added a comment -

          Attached trivial patch as discussed. I agree that we should fix this in jclouds sooner or later.

          Show
          Andrei Savu added a comment - Attached trivial patch as discussed. I agree that we should fix this in jclouds sooner or later.
          Andrei Savu made changes -
          Attachment WHIRR-330.patch [ 12482665 ]
          Andrei Savu made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Assignee Andrei Savu [ savu.andrei ]
          Fix Version/s 0.6.0 [ 12316468 ]
          Hide
          Tom White added a comment -

          +1 looks good.

          Show
          Tom White added a comment - +1 looks good.
          Hide
          Andrei Savu added a comment -

          I've just committed this. Thanks John for the extended bug description.

          Show
          Andrei Savu added a comment - I've just committed this. Thanks John for the extended bug description.
          Andrei Savu made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Patch Available Patch Available
          10h 37m 1 Andrei Savu 15/Jun/11 14:41
          Patch Available Patch Available Resolved Resolved
          1d 6h 2m 1 Andrei Savu 16/Jun/11 20:44

            People

            • Assignee:
              Andrei Savu
              Reporter:
              John Amos
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development