[TEZ-2732] DefaultSorter throws ArrayIndex exceptions on 2047 Mb size sort buffers

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.5.0, 0.6.0, 0.7.0
• Fix Version/s: 0.7.1
• Target Version/s: 0.8.0-alpha, 0.7.1
• Hadoop Flags: Reviewed
• Assignee: Rajesh Balamohan
• Reporter: Rajesh Balamohan
• Created: 20/Aug/15 09:22
• Updated: 18/May/16 04:55
• Resolved: 21/Aug/15 21:59

Description

  kvbuffer.length = 2146435072 (2047 MB)
  Corner case: bufIndex=2026133899, kvbidx=523629312.
  distkvi = mod - i + j = 2146435072 - 2026133899 + 523629312 = 643930485
  newPos = (2026133899 + (max(.., min(643930485/2, 271128624))) (This would overflow)

Would be good to restrict the max allowed sort buffer to 1800 instead of 2047.