Details
-
Type:
Improvement
-
Status: Resolved
-
Priority:
Major
-
Resolution: Fixed
-
Affects Version/s: None
-
Fix Version/s: 0.5.0
-
Component/s: service/hadoop
-
Labels:None
Description
We can use the jclouds node metadata to find out the number of cores per node, and use it to set the mapred.tasktracker.
{map,reduce}.tasks.maximum properties.
Issue Links
- relates to
-
WHIRR-229
dynamically calculate hardware requirements
-
- Resolved
-
Activity
- All
- Comments
- Work Log
- History
- Activity
- Transitions
+1 Looks good. I think we can also do a better job at setting a default value for hadoop-mapreduce.mapred.child.java.opts=-Xmx550m.
Show
Andrei Savu
added a comment - +1 Looks good. I think we can also do a better job at setting a default value for hadoop-mapreduce.mapred.child.java.opts=-Xmx550m .
Good point. I think we can commit this one, and look at the memory options in another JIRA.
I've just committed this. I believe it's a good idea to handle the memory options as part of WHIRR-229.
Show
Andrei Savu
added a comment - I've just committed this. I believe it's a good idea to handle the memory options as part of WHIRR-229 .
This patch sets the above Hadoop properties dynamically, as well as mapred.reduce.tasks (to the number of reduce slots in the cluster).
In addition to the unit test I have run the Hadoop integration test, and tried spinning up a small cluster with larger instance sizes and observed more cluster slots.