Pig
  1. Pig
  2. PIG-2128

Generating the jar file takes a lot of time and is unnecessary when running Pig local mode

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.9.2, 0.10.0
    • Component/s: impl
    • Labels:
      None
    • Patch Info:
      Patch Available
    • Release Note:
      local mode will now skip building a jar with dependencies

      Description

      Since the Pig local mode implementation has been moved to Hadoop local it is very slow.
      We should optimize some of the steps so that it is more user friendly.
      In this case we should skip building the jar for each job as it runs in the same process.

      1. PIG-2128.patch
        1 kB
        Julien Le Dem

        Activity

        Hide
        Julien Le Dem added a comment -

        attaching PIG-2128.patch to skip building the jar when in local mode

        Show
        Julien Le Dem added a comment - attaching PIG-2128 .patch to skip building the jar when in local mode
        Hide
        Dmitriy V. Ryaboy added a comment -

        Julien, good idea.

        Does this work if external jars are registered in the script?

        Show
        Dmitriy V. Ryaboy added a comment - Julien, good idea. Does this work if external jars are registered in the script?
        Hide
        Dmitriy V. Ryaboy added a comment -

        I verified that external jars do work. It's a significant speedup. +1.

        Please change title of jira to refer to this specific optimization so we don't have 15 "local mode is slow" tickets as we keep chipping away at this .

        Show
        Dmitriy V. Ryaboy added a comment - I verified that external jars do work. It's a significant speedup. +1. Please change title of jira to refer to this specific optimization so we don't have 15 "local mode is slow" tickets as we keep chipping away at this .
        Hide
        Julien Le Dem added a comment -

        I've updated the summary and description
        I am not sure if this could break other things (like register).

        Show
        Julien Le Dem added a comment - I've updated the summary and description I am not sure if this could break other things (like register).
        Hide
        Dmitriy V. Ryaboy added a comment -

        I verified that registering jars is fine and properties are picked up (in fact, because we always register fairly sizable jars, this is a significant enough improvement that I already pushed this patch to production in our clusters). So far so good. Go ahead and commit.

        Show
        Dmitriy V. Ryaboy added a comment - I verified that registering jars is fine and properties are picked up (in fact, because we always register fairly sizable jars, this is a significant enough improvement that I already pushed this patch to production in our clusters). So far so good. Go ahead and commit.
        Hide
        Viraj Bhat added a comment -

        Can this patch be backported to Pig 0.9?
        Viraj

        Show
        Viraj Bhat added a comment - Can this patch be backported to Pig 0.9? Viraj
        Hide
        Dmitriy V. Ryaboy added a comment -

        Viraj,
        Yes, this patch applies to 0.9 and even 0.8 (possibly with a bit of elbow grease – I don't remember now)

        Was this a question or a request?

        Show
        Dmitriy V. Ryaboy added a comment - Viraj, Yes, this patch applies to 0.9 and even 0.8 (possibly with a bit of elbow grease – I don't remember now) Was this a question or a request?
        Hide
        Olga Natkovich added a comment -

        We were wondering if this change can be committed to 0.9 branch. It looks pretty benign.

        Show
        Olga Natkovich added a comment - We were wondering if this change can be committed to 0.9 branch. It looks pretty benign.
        Hide
        Dmitriy V. Ryaboy added a comment -

        Patch applied cleanly, I'm running test-commit right now and will commit assuming it passes.

        Show
        Dmitriy V. Ryaboy added a comment - Patch applied cleanly, I'm running test-commit right now and will commit assuming it passes.
        Hide
        Olga Natkovich added a comment -

        Thanks!

        Show
        Olga Natkovich added a comment - Thanks!
        Hide
        Dmitriy V. Ryaboy added a comment -

        committed to 0.9.2

        Show
        Dmitriy V. Ryaboy added a comment - committed to 0.9.2

          People

          • Assignee:
            Julien Le Dem
            Reporter:
            Julien Le Dem
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development