Pig
  1. Pig
  2. PIG-482

PERFORMANCE: Use hand crafted integer/long conversions

    Details

    • Type: Improvement Improvement
    • Status: Reopened
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      instead of Integer.valueOf() and Long.valueOf(). Initial tests showed about 8x speed up in conversion time.

        Issue Links

          Activity

          Hide
          Jonathan Coveney added a comment -

          I promise I'm not trying to engage in JIRA necromancy, but this JIRA intrigued me. What part of the codebase were your referring to? Do you think that this opportunity for optimization still exists? Has it already been done elsewhere? My quest to learn the Pig internals needs to know

          Show
          Jonathan Coveney added a comment - I promise I'm not trying to engage in JIRA necromancy, but this JIRA intrigued me. What part of the codebase were your referring to? Do you think that this opportunity for optimization still exists? Has it already been done elsewhere? My quest to learn the Pig internals needs to know
          Hide
          Thejas M Nair added a comment -

          Looks like this was addressed in PIG-1028.

          Show
          Thejas M Nair added a comment - Looks like this was addressed in PIG-1028 .
          Hide
          Dmitriy V. Ryaboy added a comment -

          I don't think this is a duplicate. The linked ticket is about switching to Long.valueOf() and Integer.valueOf() (instead of using new Long(), new Integer()). This is about using something other than Long.valueOf and Integer.valueOf.

          I'd be curious to find out what Olga meant, too...

          Show
          Dmitriy V. Ryaboy added a comment - I don't think this is a duplicate. The linked ticket is about switching to Long.valueOf() and Integer.valueOf() (instead of using new Long(), new Integer()). This is about using something other than Long.valueOf and Integer.valueOf. I'd be curious to find out what Olga meant, too...
          Hide
          Thejas M Nair added a comment -

          I don't think this is a duplicate..

          I should read more carefully!

          Not sure what Olga meant. But one possibility is that she is talking about bytearray to int/long conversion in Utf8StorageConverter (used by PigStorage and some other load functions).
          It converts bytes to int/long in an inefficient way.
          It is the equivalent of -
          Integer.valueOf(Double.valueOf(new String(byteArr)).intValue); !
          The conversion from bytes to java String is likely to be most expensive part.

          Show
          Thejas M Nair added a comment - I don't think this is a duplicate.. I should read more carefully! Not sure what Olga meant. But one possibility is that she is talking about bytearray to int/long conversion in Utf8StorageConverter (used by PigStorage and some other load functions). It converts bytes to int/long in an inefficient way. It is the equivalent of - Integer.valueOf(Double.valueOf(new String(byteArr)).intValue); ! The conversion from bytes to java String is likely to be most expensive part.
          Hide
          Olga Natkovich added a comment -

          I can't recall details right now but I think this is something that Alan experimented with so he might remember exactly what improvement he has done.

          Show
          Olga Natkovich added a comment - I can't recall details right now but I think this is something that Alan experimented with so he might remember exactly what improvement he has done.

            People

            • Assignee:
              Unassigned
              Reporter:
              Olga Natkovich
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:

                Development