Details
Description
InputSplits should be sorted by length but TableSplit does not contain real getLength implementation:
@Override
public long getLength()
This is causing us problem with scheduling - we have got jobs that are supposed to finish in limited time but they get often stuck in last mapper working on large region.
Can we implement this method ?
What is the best way ?
We were thinking about estimating size by size of files on HDFS.
We would like to get Scanner from TableSplit, use startRow, stopRow and column families to get corresponding region than computing size of HDFS for given region and column family.
Update:
This ticket was about production issue - I talked with guy who worked on this and he said our production issue was probably not directly caused by getLength() returning 0.
Attachments
Attachments
Issue Links
- supercedes
-
HBASE-10019 PerfEval mapreduce percentage complete is inaccurate
- Closed