Pig
  1. Pig
  2. PIG-2146

POStore.getSchema() returns null because of which PigOutputCommitter is not storing schema while cleanup

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.8.1, 0.9.0
    • Fix Version/s: 0.9.0
    • Component/s: None
    • Labels:
      None

      Description

      The below is my script;

      register piggybank.jar;
      a = load 'myinput' using PigStorage(',') as (f1:chararray,f2:chararray,f3:chararray);
      b = distinct a;
      c = limit b 2;
      store c into 'pss001' using org.apache.pig.piggybank.storage.PigStorageSchema();
      

      Input
      -------
      a,1,aa
      b,2,bb
      c,3,cc

      For this script , PigStorageSchema is not generating .pig_headers and .pig_schema files. While debugging I could see that storeSchema(..) method itself is not invoked.The schema object for the store is returned as null (POStore.getSchema()) because of which PigOutputCommitter is not invoking the storSchema.

      The same schema object is valid when I run it in local mode. This issue is happening for Pig 0.9 also.

      1. PIG-2146.1.patch
        5 kB
        Thejas M Nair

        Activity

        Vivek Padmanabhan created issue -
        Olga Natkovich made changes -
        Field Original Value New Value
        Fix Version/s 0.9.0 [ 12315191 ]
        Hide
        Thejas M Nair added a comment -

        PIG-2146.1.patch - Fixes the issue in MRCompiler LimitAdjuster where a POStore added for new MR operation did not have the schema set.

        Show
        Thejas M Nair added a comment - PIG-2146 .1.patch - Fixes the issue in MRCompiler LimitAdjuster where a POStore added for new MR operation did not have the schema set.
        Thejas M Nair made changes -
        Attachment PIG-2146.1.patch [ 12487067 ]
        Hide
        Thejas M Nair added a comment -

        Ran test-commit and test-patch. One of the test-commit tests - TestLoad has failures, which are unrelated to this patch.

        Show
        Thejas M Nair added a comment - Ran test-commit and test-patch. One of the test-commit tests - TestLoad has failures, which are unrelated to this patch.
        Thejas M Nair made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Daniel Dai added a comment -

        +1

        Show
        Daniel Dai added a comment - +1
        Thejas M Nair made changes -
        Assignee Thejas M Nair [ thejas ]
        Hide
        Thejas M Nair added a comment -

        Patch committed to 0.9 branch and trunk.

        Show
        Thejas M Nair added a comment - Patch committed to 0.9 branch and trunk.
        Thejas M Nair made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Olga Natkovich made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Patch Available Patch Available
        21d 10h 2m 1 Thejas M Nair 19/Jul/11 22:58
        Patch Available Patch Available Resolved Resolved
        1d 59m 1 Thejas M Nair 20/Jul/11 23:58
        Resolved Resolved Closed Closed
        14d 1h 36m 1 Olga Natkovich 04/Aug/11 01:35

          People

          • Assignee:
            Thejas M Nair
            Reporter:
            Vivek Padmanabhan
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development