Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-2689

JsonStorage fails to find schema when LimitAdjuster runs

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.10.0
    • Fix Version/s: 0.14.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Scripts that both save out data with JsonStorage and trigger the LimitAdjuster (e.g. doing an order by followed by a limit) yield the following Exception:

      java.io.IOException: Could not find schema in UDF context
      at org.apache.pig.builtin.JsonStorage.prepareToWrite(JsonStorage.java:125)
      at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.<init>(PigOutputFormat.java:125)
      at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.getRecordWriter(PigOutputFormat.java:86)
      at org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.<init>(ReduceTask.java:569)
      at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:638)
      at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:417)
      at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:260)

      This happens b/c the LimitAdjuster does not copy the signature into it's newly created POStore, and hence JsonStorage looks for the schema for a null signature.

        Attachments

        1. PIG-2689.patch
          6 kB
          Doug Daniels
        2. PIG-2689-2.patch
          14 kB
          Rohini Palaniswamy
        3. PIG-2689-2-nowhitespacechange.patch
          8 kB
          Rohini Palaniswamy

          Issue Links

            Activity

              People

              • Assignee:
                ddaniels888 Doug Daniels
                Reporter:
                ddaniels888 Doug Daniels
              • Votes:
                2 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: