Pig
  1. Pig
  2. PIG-180

BinaryStorage is implicitly used where it should not be

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.1.0
    • Component/s: None
    • Labels:
      None

      Description

      When I ran the attached Pig script, I got unexpected resulting output data. After a closer at Pig's console output, it appeared BinaryStorage is incorrectly used:

      I can't find HOD configuration for piglet, hopefully you weren't planning on using HOD.
      2008-04-02 15:39:11,668 [main] INFO  org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: wilbur11.labs.corp.sp1.yahoo.com:8020
      2008-04-02 15:39:12,478 [main] INFO  org.apache.pig.backend.hadoop.executionengine.POMapreduce - ----- MapReduce Job -----
      2008-04-02 15:39:12,478 [main] INFO  org.apache.pig.backend.hadoop.executionengine.POMapreduce - Input: [/user/pig/tests/data/singlefile/studenttab10k:org.apache.pig.builtin.PigStorage()]
      2008-04-02 15:39:12,478 [main] INFO  org.apache.pig.backend.hadoop.executionengine.POMapreduce - Map: [[*]->org.apache.pig.impl.eval.StreamSpec@181edf4]
      2008-04-02 15:39:12,479 [main] INFO  org.apache.pig.backend.hadoop.executionengine.POMapreduce - Group: null
      2008-04-02 15:39:12,479 [main] INFO  org.apache.pig.backend.hadoop.executionengine.POMapreduce - Combine: null
      2008-04-02 15:39:12,479 [main] INFO  org.apache.pig.backend.hadoop.executionengine.POMapreduce - Reduce: null
      2008-04-02 15:39:12,479 [main] INFO  org.apache.pig.backend.hadoop.executionengine.POMapreduce - Output: results_7:org.apache.pig.builtin.BinaryStorage
      2008-04-02 15:39:12,479 [main] INFO  org.apache.pig.backend.hadoop.executionengine.POMapreduce - Split: null
      2008-04-02 15:39:12,480 [main] INFO  org.apache.pig.backend.hadoop.executionengine.POMapreduce - Map parallelism: -1
      2008-04-02 15:39:12,480 [main] INFO  org.apache.pig.backend.hadoop.executionengine.POMapreduce - Reduce parallelism: -1
      219190 hdfs://wilbur11.labs.corp.sp1.yahoo.com:8020/user/pig/tests/data/singlefile/studenttab10k
      2008-04-02 15:39:15,308 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Pig progress = 0%
      2008-04-02 15:39:20,329 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Pig progress = 50%
      2008-04-02 15:39:26,355 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Pig progress = 60%
      2008-04-02 15:39:28,395 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Pig progress = 80%
      2008-04-02 15:39:30,407 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Pig progress = 90%
      2008-04-02 15:39:32,415 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Pig progress = 100%
      
      1. streaming.pl
        0.1 kB
        Xu Zhang
      2. script.pig
        0.2 kB
        Xu Zhang
      3. PIG-180_0_20080402.patch
        6 kB
        Arun C Murthy

        Activity

        Hide
        Arun C Murthy added a comment -

        This patch fixes a bug in the

        {Load|Store}Optimizer where the input/output {Load|Store}

        Func of the StreamingCommand wasn't being set correctly in some cases when it got flipped to BinaryStorage.

        Show
        Arun C Murthy added a comment - This patch fixes a bug in the {Load|Store}Optimizer where the input/output {Load|Store} Func of the StreamingCommand wasn't being set correctly in some cases when it got flipped to BinaryStorage.
        Hide
        Xu Zhang added a comment -

        +1 as I saw Arun's new pig.jar work.

        Show
        Xu Zhang added a comment - +1 as I saw Arun's new pig.jar work.
        Hide
        Olga Natkovich added a comment -

        all tests passed. Patch committed. Thanks Arun for contributing and Xu for testing

        Show
        Olga Natkovich added a comment - all tests passed. Patch committed. Thanks Arun for contributing and Xu for testing

          People

          • Assignee:
            Arun C Murthy
            Reporter:
            Xu Zhang
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development