Uploaded image for project: 'ORC'
  1. ORC
  2. ORC-476

Make SearchAgument kryo buffer size configurable

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.4.5, 1.5.5, 1.6.0
    • Component/s: MapReduce
    • Labels:
      None

      Description

      The kryo output buffer size for SArg is hardcoded right now to 100000. The hive implementation sets this to an initial size of 4096 and max size to 10485760.

       

      Spark started using the apache orc implementation from 2.3 compared to the hive implementation. Spark jobs are failing with buffer overflow error due to the small size of the buffer here. We should make this configurable so that frameworks using the orc implementation can pass the size configs.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Dhruve Ashar Dhruve Ashar
                Reporter:
                Dhruve Ashar Dhruve Ashar
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 20m
                  20m