|
If you were logged in you would be able to see more operations.
|
|
|
| Hadoop Flags: |
Reviewed
|
| Resolution Date: |
25/Nov/08 10:38 PM
|
|
For each pass of the sort and merge, I would like a count of the number of records. So for example, if the map output 100 records and they were sorted once, the counter would be 100. If it spilled twice and was merged together, it would be 200. Clearly in a multi-level merge, it may not be a multiple of the number of map output records. This would let the users easily see if they have values like io.sort.mb or io.sort.factor set too low.
|
|
Description
|
For each pass of the sort and merge, I would like a count of the number of records. So for example, if the map output 100 records and they were sorted once, the counter would be 100. If it spilled twice and was merged together, it would be 200. Clearly in a multi-level merge, it may not be a multiple of the number of map output records. This would let the users easily see if they have values like io.sort.mb or io.sort.factor set too low. |
Show » |
| No work has yet been logged on this issue.
|
|
Improvement
Closed
Major
Add counters to show number of key/values that have been sorted and merged in the maps and reduces
