Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-4059 Pig on Spark
  3. PIG-4239

"pig.output.lazy" not works in spark mode

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: spark-branch
    • Component/s: spark
    • Labels:
      None

      Description

      If "pig.output.lazy" is "true", empty part files from the output will be omitted.
      steps to reproduce
      1. set "pig.output.lazy" as “true" in $PIG_HOME/conf/pig.properties.
      2. run following lazy.pig script in spark mode:
      cat lazy.pig
      a = load '/user/pig/lazy' using PigStorage();
      b = filter a by $0 == 'hey';
      c = store b into '/tmp/lazy.out';
      lazy.pig and lazy are attached
      3. empty file "/tmp/lazy.out/part-rxxxx” is still generated, the empty file is expected not to be generated when "pig.output.lazy" is "true"

        Attachments

        1. lazy
          0.0 kB
          liyunzhang
        2. lazy.pig
          0.1 kB
          liyunzhang
        3. PIG-4239_1.patch
          3 kB
          liyunzhang
        4. PIG-4239.patch
          2 kB
          liyunzhang

          Issue Links

            Activity

              People

              • Assignee:
                kellyzly liyunzhang
                Reporter:
                kellyzly liyunzhang
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: