Uploaded image for project: 'Apache Tez'
  1. Apache Tez
  2. TEZ-3202

Reduce the memory need for jobs with high number of segments

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 0.7.1, 0.8.4
    • None
    • None

    Description

      Segment has a 'key' member that holds accounting information to the reader's current key buffer, position, and length. There is a 384 byte overhead per segment since the account is done with the DataInputBuffer class which derives from DataInputStream which has underlying byte[80] and char[80] among significant pieces. This jira aims to reduce the overhead per segment

      Attachments

        1. TEZ-3202.1.patch
          9 kB
          Jonathan Turner Eagles
        2. TEZ-3202.2.patch
          24 kB
          Jonathan Turner Eagles
        3. TEZ-3202.3.patch
          24 kB
          Jonathan Turner Eagles
        4. TEZ-3202.4.patch
          22 kB
          Jonathan Turner Eagles
        5. TEZ-3202.4-branch-0.7.patch
          22 kB
          Jonathan Turner Eagles

        Activity

          People

            jeagles Jonathan Turner Eagles
            jeagles Jonathan Turner Eagles
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: