Details
-
Improvement
-
Status: Open
-
Major
-
Resolution: Unresolved
-
None
-
None
-
None
-
None
Description
Any time we chain together M-R jobs, we doing it because we need separate reducer like with group by followed by order by. We don't really need the maps. The ideal graph for us would be:
M->R->SortShuffle->R->SortShuffle->R ...
This would allow us to save read from DFs and write to the local disk which could be fairly significant.
Aparently this similar discussion took place on hadoop mailing list several times and this request was turned down. Main reason is that in their opinion cost of implementing something like that would outweigh the benefit.
To make a persuasive case, we need to measure the overhead of the empty maps for "typical" queries.