Pig
  1. Pig
  2. PIG-2702

Make Pig local mode (and tests) faster by working around the hard coded sleep(5000) in hadoop's JobControl

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.11
    • Component/s: None
    • Labels:
      None
    • Patch Info:
      Patch Available

      Description

      the JobControl class polls the jobs statuses with a hard coded sleep(5000) making every job take a minimum of 5 seconds.
      Since the local mode uses the same code path this makes local mode (and unit tests) extremely slow.

      1. PIG-2702_a.patch
        23 kB
        Julien Le Dem
      2. PIG-2702.patch
        20 kB
        Julien Le Dem

        Activity

        Julien Le Dem created issue -
        Julien Le Dem made changes -
        Field Original Value New Value
        Summary Make Pig local mode (and tests) faster by removing the hard coded sleep(5000) in hadoopJobControl Make Pig local mode (and tests) faster by working around the hard coded sleep(5000) in hadoop's JobControl
        Description the JobControl class polls the jobs statuses with a hard coded sleep(5000) making every job take a minimum of 5 seconds.
        Since the local mode uses the same code path this makes local mode (and unit tests) extremely slow.
        Julien Le Dem made changes -
        Assignee Julien Le Dem [ julienledem ]
        Julien Le Dem made changes -
        Attachment PIG-2702.patch [ 12527460 ]
        Julien Le Dem made changes -
        Patch Info Patch Available [ 10042 ]
        Julien Le Dem made changes -
        Attachment PIG-2702_a.patch [ 12527487 ]
        Julien Le Dem made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Fix Version/s 0.11 [ 12318878 ]
        Resolution Fixed [ 1 ]
        Bill Graham made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            Julien Le Dem
            Reporter:
            Julien Le Dem
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development