Whirr
  1. Whirr
  2. WHIRR-282

Set number of Hadoop slots based on hardware

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.5.0
    • Component/s: service/hadoop
    • Labels:
      None

      Description

      We can use the jclouds node metadata to find out the number of cores per node, and use it to set the mapred.tasktracker.

      {map,reduce}

      .tasks.maximum properties.

        Issue Links

          Activity

          Hide
          Tom White added a comment -

          This patch sets the above Hadoop properties dynamically, as well as mapred.reduce.tasks (to the number of reduce slots in the cluster).

          In addition to the unit test I have run the Hadoop integration test, and tried spinning up a small cluster with larger instance sizes and observed more cluster slots.

          Show
          Tom White added a comment - This patch sets the above Hadoop properties dynamically, as well as mapred.reduce.tasks (to the number of reduce slots in the cluster). In addition to the unit test I have run the Hadoop integration test, and tried spinning up a small cluster with larger instance sizes and observed more cluster slots.
          Hide
          Andrei Savu added a comment -

          +1 Looks good. I think we can also do a better job at setting a default value for hadoop-mapreduce.mapred.child.java.opts=-Xmx550m.

          Show
          Andrei Savu added a comment - +1 Looks good. I think we can also do a better job at setting a default value for hadoop-mapreduce.mapred.child.java.opts=-Xmx550m .
          Hide
          Tom White added a comment -

          Good point. I think we can commit this one, and look at the memory options in another JIRA.

          Show
          Tom White added a comment - Good point. I think we can commit this one, and look at the memory options in another JIRA.
          Hide
          Andrei Savu added a comment -

          I've just committed this. I believe it's a good idea to handle the memory options as part of WHIRR-229.

          Show
          Andrei Savu added a comment - I've just committed this. I believe it's a good idea to handle the memory options as part of WHIRR-229 .

            People

            • Assignee:
              Tom White
              Reporter:
              Tom White
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development