Whirr
  1. Whirr
  2. WHIRR-282

Set number of Hadoop slots based on hardware

        Details

        • Type: Improvement Improvement
        • Status: Resolved
        • Priority: Major Major
        • Resolution: Fixed
        • Affects Version/s: None
        • Fix Version/s: 0.5.0
        • Component/s: service/hadoop
        • Labels:
          None

          Description

          We can use the jclouds node metadata to find out the number of cores per node, and use it to set the mapred.tasktracker.

          {map,reduce}

          .tasks.maximum properties.

            Issue Links

            relates to

            Improvement - An improvement or enhancement to an existing feature or task. WHIRR-229 dynamically calculate hardware requirements

            • Major - Major loss of function.
            • Resolved

              Activity

              Ascending order - Click to sort in descending order
              Tom White created issue -
              Hide
              Permalink
              Tom White added a comment -

              This patch sets the above Hadoop properties dynamically, as well as mapred.reduce.tasks (to the number of reduce slots in the cluster).

              In addition to the unit test I have run the Hadoop integration test, and tried spinning up a small cluster with larger instance sizes and observed more cluster slots.

              Show
              Tom White added a comment - This patch sets the above Hadoop properties dynamically, as well as mapred.reduce.tasks (to the number of reduce slots in the cluster). In addition to the unit test I have run the Hadoop integration test, and tried spinning up a small cluster with larger instance sizes and observed more cluster slots.
              Tom White made changes -
              Field Original Value New Value
              Attachment WHIRR-282.patch [ 12476289 ]
              Hide
              Permalink
              Andrei Savu added a comment -

              +1 Looks good. I think we can also do a better job at setting a default value for hadoop-mapreduce.mapred.child.java.opts=-Xmx550m.

              Show
              Andrei Savu added a comment - +1 Looks good. I think we can also do a better job at setting a default value for hadoop-mapreduce.mapred.child.java.opts=-Xmx550m .
              Hide
              Permalink
              Tom White added a comment -

              Good point. I think we can commit this one, and look at the memory options in another JIRA.

              Show
              Tom White added a comment - Good point. I think we can commit this one, and look at the memory options in another JIRA.
              Hide
              Permalink
              Andrei Savu added a comment -

              I've just committed this. I believe it's a good idea to handle the memory options as part of WHIRR-229.

              Show
              Andrei Savu added a comment - I've just committed this. I believe it's a good idea to handle the memory options as part of WHIRR-229 .
              Andrei Savu made changes -
              Status Open [ 1 ] Resolved [ 5 ]
              Resolution Fixed [ 1 ]
              Andrei Savu made changes -
              Link This issue relates to WHIRR-229 [ WHIRR-229 ]

                People

                • Assignee:
                  Tom White
                  Reporter:
                  Tom White
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  0 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved:

                    Development