Pig
  1. Pig
  2. PIG-2128

Generating the jar file takes a lot of time and is unnecessary when running Pig local mode

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.9.2, 0.10.0
    • Component/s: impl
    • Labels:
      None
    • Patch Info:
      Patch Available
    • Release Note:
      local mode will now skip building a jar with dependencies

      Description

      Since the Pig local mode implementation has been moved to Hadoop local it is very slow.
      We should optimize some of the steps so that it is more user friendly.
      In this case we should skip building the jar for each job as it runs in the same process.

      1. PIG-2128.patch
        1 kB
        Julien Le Dem

        Activity

        Julien Le Dem created issue -
        Hide
        Julien Le Dem added a comment -

        attaching PIG-2128.patch to skip building the jar when in local mode

        Show
        Julien Le Dem added a comment - attaching PIG-2128 .patch to skip building the jar when in local mode
        Julien Le Dem made changes -
        Field Original Value New Value
        Attachment PIG-2128.patch [ 12482714 ]
        Julien Le Dem made changes -
        Assignee Julien Le Dem [ julienledem ]
        Hide
        Dmitriy V. Ryaboy added a comment -

        Julien, good idea.

        Does this work if external jars are registered in the script?

        Show
        Dmitriy V. Ryaboy added a comment - Julien, good idea. Does this work if external jars are registered in the script?
        Hide
        Dmitriy V. Ryaboy added a comment -

        I verified that external jars do work. It's a significant speedup. +1.

        Please change title of jira to refer to this specific optimization so we don't have 15 "local mode is slow" tickets as we keep chipping away at this .

        Show
        Dmitriy V. Ryaboy added a comment - I verified that external jars do work. It's a significant speedup. +1. Please change title of jira to refer to this specific optimization so we don't have 15 "local mode is slow" tickets as we keep chipping away at this .
        Hide
        Julien Le Dem added a comment -

        I've updated the summary and description
        I am not sure if this could break other things (like register).

        Show
        Julien Le Dem added a comment - I've updated the summary and description I am not sure if this could break other things (like register).
        Julien Le Dem made changes -
        Summary Pig local mode is very slow. Generating the jar file takes a lot of time and is unnecessary when running Pig local mode
        Description Since the Pig local mode implementation has been moved to Hadoop local it is very slow.
        We should optimize some of the steps so that it is more user friendly.
        One thing would be to skip building the jar for each job as it runs in the same process.
        Since the Pig local mode implementation has been moved to Hadoop local it is very slow.
        We should optimize some of the steps so that it is more user friendly.
        In this case we should skip building the jar for each job as it runs in the same process.
        Hide
        Dmitriy V. Ryaboy added a comment -

        I verified that registering jars is fine and properties are picked up (in fact, because we always register fairly sizable jars, this is a significant enough improvement that I already pushed this patch to production in our clusters). So far so good. Go ahead and commit.

        Show
        Dmitriy V. Ryaboy added a comment - I verified that registering jars is fine and properties are picked up (in fact, because we always register fairly sizable jars, this is a significant enough improvement that I already pushed this patch to production in our clusters). So far so good. Go ahead and commit.
        Alan Gates made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Julien Le Dem made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Release Note local mode will now skip building a jar with dependencies
        Fix Version/s 0.10 [ 12316246 ]
        Resolution Fixed [ 1 ]
        Hide
        Viraj Bhat added a comment -

        Can this patch be backported to Pig 0.9?
        Viraj

        Show
        Viraj Bhat added a comment - Can this patch be backported to Pig 0.9? Viraj
        Hide
        Dmitriy V. Ryaboy added a comment -

        Viraj,
        Yes, this patch applies to 0.9 and even 0.8 (possibly with a bit of elbow grease – I don't remember now)

        Was this a question or a request?

        Show
        Dmitriy V. Ryaboy added a comment - Viraj, Yes, this patch applies to 0.9 and even 0.8 (possibly with a bit of elbow grease – I don't remember now) Was this a question or a request?
        Hide
        Olga Natkovich added a comment -

        We were wondering if this change can be committed to 0.9 branch. It looks pretty benign.

        Show
        Olga Natkovich added a comment - We were wondering if this change can be committed to 0.9 branch. It looks pretty benign.
        Hide
        Dmitriy V. Ryaboy added a comment -

        Patch applied cleanly, I'm running test-commit right now and will commit assuming it passes.

        Show
        Dmitriy V. Ryaboy added a comment - Patch applied cleanly, I'm running test-commit right now and will commit assuming it passes.
        Hide
        Olga Natkovich added a comment -

        Thanks!

        Show
        Olga Natkovich added a comment - Thanks!
        Dmitriy V. Ryaboy made changes -
        Fix Version/s 0.9.2 [ 12318248 ]
        Hide
        Dmitriy V. Ryaboy added a comment -

        committed to 0.9.2

        Show
        Dmitriy V. Ryaboy added a comment - committed to 0.9.2
        Daniel Dai made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Patch Available Patch Available
        27d 23h 34m 1 Alan Gates 13/Jul/11 21:26
        Patch Available Patch Available Resolved Resolved
        1h 12m 1 Julien Le Dem 13/Jul/11 22:38
        Resolved Resolved Closed Closed
        193d 9h 52m 1 Daniel Dai 23/Jan/12 07:31

          People

          • Assignee:
            Julien Le Dem
            Reporter:
            Julien Le Dem
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development