Pig
  1. Pig
  2. PIG-3567

LogicalPlanPrinter throws OOM for large scripts

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.12.1, 0.13.0
    • Component/s: None
    • Labels:
      None

      Description

      As mentioned in PIG-3455, LogicalPlanPrinter throws OOM for large scripts. Problem is LogicalPlanPrinter's visit method generates a large string before its written to the PrintStream.

      1. PIG-3567.patch
        10 kB
        Aniket Mokashi

        Activity

        Aniket Mokashi created issue -
        Aniket Mokashi made changes -
        Field Original Value New Value
        Assignee Aniket Mokashi [ aniket486 ]
        Aniket Mokashi made changes -
        Attachment PIG-3567.patch [ 12613774 ]
        Hide
        Aniket Mokashi added a comment -

        Fixed it by making LogicalPlanPrinter write to stream recursively (patch attached). There are no existing tests for verifying printing for logical plan. Do we need to add tests for this(changes are non-trivial)? I verified with a few scripts that logical plan remains the same.

        Show
        Aniket Mokashi added a comment - Fixed it by making LogicalPlanPrinter write to stream recursively (patch attached). There are no existing tests for verifying printing for logical plan. Do we need to add tests for this(changes are non-trivial)? I verified with a few scripts that logical plan remains the same.
        Aniket Mokashi made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Aniket Mokashi added a comment -

        Committed to trunk and branch-0.12.

        Thanks Daniel Dai for the review.

        Show
        Aniket Mokashi added a comment - Committed to trunk and branch-0.12. Thanks Daniel Dai for the review.
        Aniket Mokashi made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Aniket Mokashi made changes -
        Fix Version/s 0.12.1 [ 12324970 ]
        Fix Version/s 0.13.0 [ 12324971 ]
        Prashant Kommireddi made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Hide
        Skanda Prasad added a comment -

        I'm now getting the following exception after applying the fix:

        Pig Stack Trace
        ---------------
        ERROR 2998: Unhandled internal error. Java heap space

        java.lang.OutOfMemoryError: Java heap space
        at java.util.Arrays.copyOf(Arrays.java:2271)
        at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:113)
        at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
        at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:140)
        at java.io.PrintStream.write(PrintStream.java:480)
        at java.io.FilterOutputStream.write(FilterOutputStream.java:97)
        at org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.writeWithTabs(LogicalPlanPrinter.java:103)
        at org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.depthFirst(LogicalPlanPrinter.java:125)
        at org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.depthFirst(LogicalPlanPrinter.java:133)
        at org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.depthFirst(LogicalPlanPrinter.java:133)
        at org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.depthFirst(LogicalPlanPrinter.java:133)
        at org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.depthFirst(LogicalPlanPrinter.java:133)
        rg.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.writeWithTabs
        at org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.depthFirst(LogicalPlanPrinter.java:133)
        at org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.depthFirst(LogicalPlanPrinter.java:133)
        at org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.depthFirst(LogicalPlanPrinter.java:133)
        at org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.depthFirst(LogicalPlanPrinter.java:133)
        at org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.depthFirst(LogicalPlanPrinter.java:133)
        at org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.depthFirst(LogicalPlanPrinter.java:133)
        at org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.depthFirst(LogicalPlanPrinter.java:133)
        at org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.depthFirst(LogicalPlanPrinter.java:133)
        at org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.depthFirst(LogicalPlanPrinter.java:133)
        at org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.depthFirst(LogicalPlanPrinter.java:133)
        at org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.depthFirst(LogicalPlanPrinter.java:133)
        at org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.depthFirst(LogicalPlanPrinter.java:133)
        at org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.depthFirst(LogicalPlanPrinter.java:133)
        at org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.depthFirst(LogicalPlanPrinter.java:133)
        at org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.depthFirst(LogicalPlanPrinter.java:133)
        at org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.depthFirst(LogicalPlanPrinter.java:133)
        at org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.depthFirst(LogicalPlanPrinter.java:133)
        at org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.depthFirst(LogicalPlanPrinter.java:133)
        at org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.depthFirst(LogicalPlanPrinter.java:133)
        at org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.depthFirstLP(LogicalPlanPrinter.java:97)

        Show
        Skanda Prasad added a comment - I'm now getting the following exception after applying the fix: Pig Stack Trace --------------- ERROR 2998: Unhandled internal error. Java heap space java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOf(Arrays.java:2271) at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:113) at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93) at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:140) at java.io.PrintStream.write(PrintStream.java:480) at java.io.FilterOutputStream.write(FilterOutputStream.java:97) at org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.writeWithTabs(LogicalPlanPrinter.java:103) at org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.depthFirst(LogicalPlanPrinter.java:125) at org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.depthFirst(LogicalPlanPrinter.java:133) at org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.depthFirst(LogicalPlanPrinter.java:133) at org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.depthFirst(LogicalPlanPrinter.java:133) at org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.depthFirst(LogicalPlanPrinter.java:133) rg.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.writeWithTabs at org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.depthFirst(LogicalPlanPrinter.java:133) at org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.depthFirst(LogicalPlanPrinter.java:133) at org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.depthFirst(LogicalPlanPrinter.java:133) at org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.depthFirst(LogicalPlanPrinter.java:133) at org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.depthFirst(LogicalPlanPrinter.java:133) at org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.depthFirst(LogicalPlanPrinter.java:133) at org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.depthFirst(LogicalPlanPrinter.java:133) at org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.depthFirst(LogicalPlanPrinter.java:133) at org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.depthFirst(LogicalPlanPrinter.java:133) at org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.depthFirst(LogicalPlanPrinter.java:133) at org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.depthFirst(LogicalPlanPrinter.java:133) at org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.depthFirst(LogicalPlanPrinter.java:133) at org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.depthFirst(LogicalPlanPrinter.java:133) at org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.depthFirst(LogicalPlanPrinter.java:133) at org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.depthFirst(LogicalPlanPrinter.java:133) at org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.depthFirst(LogicalPlanPrinter.java:133) at org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.depthFirst(LogicalPlanPrinter.java:133) at org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.depthFirst(LogicalPlanPrinter.java:133) at org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.depthFirst(LogicalPlanPrinter.java:133) at org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.depthFirstLP(LogicalPlanPrinter.java:97)
        Hide
        Skanda Prasad added a comment -

        Opened PIG-4317

        Show
        Skanda Prasad added a comment - Opened PIG-4317
        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Patch Available Patch Available
        2d 1h 38m 1 Aniket Mokashi 14/Nov/13 03:11
        Patch Available Patch Available Resolved Resolved
        12d 17h 9m 1 Aniket Mokashi 26/Nov/13 20:20
        Resolved Resolved Closed Closed
        140d 24m 1 Prashant Kommireddi 15/Apr/14 21:44

          People

          • Assignee:
            Aniket Mokashi
            Reporter:
            Aniket Mokashi
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development