Pig
  1. Pig
  2. PIG-250

Pig is broken with speculative execution

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.1.0
    • Component/s: None
    • Labels:
      None

      Description

      If I have speculative execution turned on, the following script fails:

      a = load 'studenttab20m' as (name, age, gpa);
      b = load 'votertab10k' as (name, age, registration, contributions);
      c = filter a by age < '50';
      d = filter b by age < '50';
      e = cogroup c by (name, age), d by (name, age) parallel 10;
      f = foreach e generate flatten(c), flatten(d) parallel 10;
      g = group f by registration parallel 10;
      h = foreach g generate group, SUM(f.d::contributions) parallel 10;
      i = order h by ($1, $0);
      store i into 'out';

      I traced this to the fact that the first MR job produces one or more empty outputs from the reducer. This happened on the reducers that happened to have second task running.

      I am not sure what the issue is and I am working with hadoop guys to investigate. Until this issue is resolved, I would like to trun speculative execution off.

      1. PIG-250_v2.patch
        2 kB
        Olga Natkovich
      2. PIG-250.patch
        1 kB
        Olga Natkovich

        Activity

        Alan Gates made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Alan Gates made changes -
        Assignee Olga Natkovich [ olgan ]
        Olga Natkovich made changes -
        Fix Version/s 0.1.0 [ 12312848 ]
        Olga Natkovich made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Olga Natkovich made changes -
        Attachment PIG-250_v2.patch [ 12383032 ]
        Olga Natkovich made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Olga Natkovich made changes -
        Field Original Value New Value
        Attachment PIG-250.patch [ 12382893 ]
        Olga Natkovich created issue -

          People

          • Assignee:
            Olga Natkovich
            Reporter:
            Olga Natkovich
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development